Your Speech, Their Rules: Meet the People Who Guard the Internet

Tech platform trust and safety employees are charged with policing the impossible. They open up to Medium’s head of trust and safety.

Creators and product people want to live in optimism, in an idealized vision of how people will use the product, not the ways that people will predictably break it. They just don’t want to live there. The separation of product people and trust people worries me, because in a world where product managers and engineers and visionaries cared about this stuff, it would be baked into how things get built. If things stay this way—that product and engineering are Mozart and everyone else is Alfred the butler—the big stuff is not going to change..

How has this job affected you personally?

Nora: We have executives and product managers shadow trust and safety agents during calls with users. Sitting with an agent talking to a sexual assault victim helps build some empathy, so when they go back to their teams, it’s running in the back of their brain when they’re thinking about building things. Then, when we go into a room and say, “If you do that, you’re going to increase human trafficking,” they don’t think we’re just making it up.

Jessica: Early on at YouTube, a psychologist or therapist or whatever came in. They were really inexperienced with this sort of stuff. There was this moment where everyone was telling their stories about all of the most horrific things they saw. Their goal was, “Can we gross this person out?” That therapist never came back. It was like, “Great. Not helpful, everyone.”

 

Adam: I feel like I carry around the knowledge of how easily a person’s life can be destroyed. You might make a tasteless joke, or someone might accuse you of something or want to discredit you, and then a mob of millions can converge within hours to hate you and stalk you and lie about you and encourage unstable lunatics to kill you. It can happen to any of us. You will be torn to shreds. You’re one of the eggs broken to make the free speech omelet. Your life will never be the same. The internet will move on to the next thing. And the free speech absolutists and the disingenuous trolls will shrug together and say, “Gee, I guess some snowflakes just can’t handle hearing views they disagree with.”

Y.X. Yang: You kind of have to quarantine the team in a separate part of the building, because they have a very particular sense of humor that nobody else will understand. That kind of humor is very necessary, because everyone needs a safety valve. We say really weird things and look at really weird things. You think you have the stomach to see the really weird things, but I guarantee you, you do not. Your weird is maybe some kind of strange fetish that you saw on Reddit, but our weird is three orders of magnitude beyond.

Remy: If you look at a weight lifter, their body shape becomes a certain way because they’re putting themself under this strain and stress again and again. What’s the equivalent sort of psychological or mental adaptation that a trust and safety person ends up developing? You’re kind of constantly sharpening the knife of your judgment. All the time you’re like, “Okay, we’re going to just do the right decision in the right moment in the right situation in the right context.” And at the same time, you’re also using that sharpened tool to, like, open cans and do stuff you shouldn’t. You’re constantly sharpening and blunting that blade at the same time.

Mathilda: It’s actually made me more sympathetic to people and conscious of my own actions, which is the opposite of what I thought sifting through hatred and abuse all day would do. The internet divvies up power in strange ways that have many democratic benefits but that also allow suffering and shame to be amplified. It’s really made me change my tone when I send emails to other support teams, because I know what will make someone go above and beyond to help me and what will be passed around the team as a joke.

.. What about the working conditions and pay among people doing this work?

Remy: It only makes sense that whatever negative and toxic effects ordinary people get from heavy internet use will only be multiplied in people whose job is to deal with the worst of it. It’s a no-brainer. I don’t know what the solution should be. The only option I really see people propose is “magic A.I.,” but we still have to train and check automated systems (never mind build them!). So, how would that solve it? It just pushes the burden to someone else.

Mathilda: This happens in lots of industries. A company makes something and externalizes costs that others have to pay. Partly because that’s what current corporate principles incentivize, partly because when you make a new thing, no one has any idea what the real long-term costs are. People post millions of things a day on dozens of platforms. And they’re demanding more moderation. How? It can be done by people or robots or a combo. The robots are not good at it yet and may never be, especially if you like free speech. And people are expensive, even when you don’t pay them well. I think if you really seriously internalized the long-run, environmental costs of platforms as they exist today, we’d be blown away by the cost. But it’s impossible to ignore. So, I think the companies themselves should be more responsible for internalizing that cost or modifying their business models. And if the money’s already been taken out of the company by a relative few who received capital windfalls, then they (and their investors and bankers) should give it back so we can pay moderators a fair wage. This ecological disaster made them billions. Let them pay for the cleanup.

 

.. My partner has been like, “Hey, when we’re in public, could you say SA instead of sexual assault, because it’s really not cool when we’re in public and you’re talking openly about sexual assault and people overhear it. It’s just not fun for me.” I was like, I don’t even think about it, because it’s all I do all day. Some people are like, “What’s the worst thing that’s ever happened?” And I have learned the hard way, you think you want to know the worst thing that’s ever happened, but you don’t.

Welcome Wagon: Classifying Comments on Stack Overflow

I (Jason) wrote The Stack Overflow Comment Evaluator 5000™, a simple application that presents you with a comment thread from a post on Stack Overflow and asks you to rate each comment in the thread as Fine, Unwelcoming, or Abusive.

Prevalence of comment categories

If we take a majority vote on the rating of each comment (with ties going to the worse rating) comments on Stack Overflow break down like so…

Rating % of comments
Fine 92.3%
Unwelcoming 7.4%
Abusive 0.3%

 

According to those of us deeply involved here and familiar with Stack Overflow, about 7% of comments on Stack Overflow are unwelcoming. What did some unwelcoming comments look like? These combine elements of real comments to show typical examples.

  • “This is becoming a waste of my time and you won’t listen to my advice. What are the supposed benefits of making it so much more complex?”
  • “Step 1. Do not clutter the namespace. Then get back to us.”
  • “The code you posted cannot yield this result. Please post the real code if you hope to get any help.”
  • “This error is self explanatory. You need to check…”
  • “I have already told how you can… If you can’t make it work, you are doing something wrong.”

This stuff isn’t profane, hate, or outright abuse, but it’s certainly unwelcoming. Looking at majority voting is one approach, but the experience of being not welcomed is not a majority vote kind of thing; it’s deeply personal. What if we looked at the distributions of the ratings by individual?

 

 

.. Firstly the “unwelcoming” comments aren’t unwelcoming. They’re all valid criticisms, I imagine. You haven’t given us the context in which they’re said, which’d be extremely helpful here.

The abusive comments make up 1 in 250. Which is tiny. That’s such a small amount that I find it hard to believe you’re even worrying about it. It’s -genuinely- impressive that it’s so much lower than a great deal of other websites that seek to achieve the same thing as this one. You should be extremely proud of this. It’s never going to be perfect.

I love this site and I love what it does. It’s genuinely a great platform for doing what it does. But you should work on improving the parts of it that are lacking (like chat!) before you try to handle such a minutely small problem.

 

.. The unwelcoming comments are valid criticisms, that can be expressed in a much more welcoming manner. The problem is large enough that StackOverflow is stereotyped and memed as being unwelcoming. That warrants attention, in my book.

.. I think you’re coming at this from the wrong angle. You’re thinking “let’s calculate a metric, and if that metric is below X, we don’t have a problem. Everyone who thinks there is a significant problem is wrong.”

Whereas I think the reality is: a huge number of people think there is a problem. Women in particular, who don’t contribute because they think it’s an unwelcoming place. So I’d reframe it as “let’s calculate a metric X, and now we know that at level X, we have a problem. We don’t yet know at what level we don’t have a problem anymore, but it must be less than X’.

 

.. SO claims to be a Q/A platform for professional and enthusiast programmers and for questions about programming that are tightly focused on a specific problem. So the questions must exactly tell that specific problem. This is often nothing what a beginner can do. So in my opinion SO is not a Q/A platform for beginners in programming. So this user group will always feel not welcoming here simply because it is not. Maybe SO should providing a special beginners Q/A portal additionally?

 

.. Maybe my personal bias is showing, but these 5 example “unwelcoming” comments don’t look like that to me at all. I’ve definitely gotten way harsher ones on my posts in the past and at no point did I feel them to be unwelcoming whatsoever.

And leaving these up so readers can identify the users leaving them, or even completly citing them so they can be easily found in SEDE is a privacy violation in my opinion and nobody deserves to be put in a hall of shame like that, especially not on stackexchange where the standing policy has been to allow people to correct their misgivings in private. Disappointing.

 

.. As it was stated above, those examples of unwelcoming comments above weren’t actual comments pulled from SO; they were pieced together from little bits and pieces of unwelcome comments. You can’t identify the writers of the original comments, as they weren’t written by a user. There is no privacy violation. No one has been put in a hall of shame.