Tech platform trust and safety employees are charged with policing the impossible. They open up to Medium’s head of trust and safety.
Creators and product people want to live in optimism, in an idealized vision of how people will use the product, not the ways that people will predictably break it. They just don’t want to live there. The separation of product people and trust people worries me, because in a world where product managers and engineers and visionaries cared about this stuff, it would be baked into how things get built. If things stay this way—that product and engineering are Mozart and everyone else is Alfred the butler—the big stuff is not going to change..
How has this job affected you personally?
Nora: We have executives and product managers shadow trust and safety agents during calls with users. Sitting with an agent talking to a sexual assault victim helps build some empathy, so when they go back to their teams, it’s running in the back of their brain when they’re thinking about building things. Then, when we go into a room and say, “If you do that, you’re going to increase human trafficking,” they don’t think we’re just making it up.
Jessica: Early on at YouTube, a psychologist or therapist or whatever came in. They were really inexperienced with this sort of stuff. There was this moment where everyone was telling their stories about all of the most horrific things they saw. Their goal was, “Can we gross this person out?” That therapist never came back. It was like, “Great. Not helpful, everyone.”
Adam: I feel like I carry around the knowledge of how easily a person’s life can be destroyed. You might make a tasteless joke, or someone might accuse you of something or want to discredit you, and then a mob of millions can converge within hours to hate you and stalk you and lie about you and encourage unstable lunatics to kill you. It can happen to any of us. You will be torn to shreds. You’re one of the eggs broken to make the free speech omelet. Your life will never be the same. The internet will move on to the next thing. And the free speech absolutists and the disingenuous trolls will shrug together and say, “Gee, I guess some snowflakes just can’t handle hearing views they disagree with.”
Y.X. Yang: You kind of have to quarantine the team in a separate part of the building, because they have a very particular sense of humor that nobody else will understand. That kind of humor is very necessary, because everyone needs a safety valve. We say really weird things and look at really weird things. You think you have the stomach to see the really weird things, but I guarantee you, you do not. Your weird is maybe some kind of strange fetish that you saw on Reddit, but our weird is three orders of magnitude beyond.
Remy: If you look at a weight lifter, their body shape becomes a certain way because they’re putting themself under this strain and stress again and again. What’s the equivalent sort of psychological or mental adaptation that a trust and safety person ends up developing? You’re kind of constantly sharpening the knife of your judgment. All the time you’re like, “Okay, we’re going to just do the right decision in the right moment in the right situation in the right context.” And at the same time, you’re also using that sharpened tool to, like, open cans and do stuff you shouldn’t. You’re constantly sharpening and blunting that blade at the same time.
Mathilda: It’s actually made me more sympathetic to people and conscious of my own actions, which is the opposite of what I thought sifting through hatred and abuse all day would do. The internet divvies up power in strange ways that have many democratic benefits but that also allow suffering and shame to be amplified. It’s really made me change my tone when I send emails to other support teams, because I know what will make someone go above and beyond to help me and what will be passed around the team as a joke.
.. What about the working conditions and pay among people doing this work?
Remy: It only makes sense that whatever negative and toxic effects ordinary people get from heavy internet use will only be multiplied in people whose job is to deal with the worst of it. It’s a no-brainer. I don’t know what the solution should be. The only option I really see people propose is “magic A.I.,” but we still have to train and check automated systems (never mind build them!). So, how would that solve it? It just pushes the burden to someone else.
Mathilda: This happens in lots of industries. A company makes something and externalizes costs that others have to pay. Partly because that’s what current corporate principles incentivize, partly because when you make a new thing, no one has any idea what the real long-term costs are. People post millions of things a day on dozens of platforms. And they’re demanding more moderation. How? It can be done by people or robots or a combo. The robots are not good at it yet and may never be, especially if you like free speech. And people are expensive, even when you don’t pay them well. I think if you really seriously internalized the long-run, environmental costs of platforms as they exist today, we’d be blown away by the cost. But it’s impossible to ignore. So, I think the companies themselves should be more responsible for internalizing that cost or modifying their business models. And if the money’s already been taken out of the company by a relative few who received capital windfalls, then they (and their investors and bankers) should give it back so we can pay moderators a fair wage. This ecological disaster made them billions. Let them pay for the cleanup.
.. My partner has been like, “Hey, when we’re in public, could you say SA instead of sexual assault, because it’s really not cool when we’re in public and you’re talking openly about sexual assault and people overhear it. It’s just not fun for me.” I was like, I don’t even think about it, because it’s all I do all day. Some people are like, “What’s the worst thing that’s ever happened?” And I have learned the hard way, you think you want to know the worst thing that’s ever happened, but you don’t.