How Bias is Built into Algorithms (w/ Cathy O’Neil)

Cathy O’Neil, CEO of O’Neil Risk Consulting & Algorithmic Auditing, explains to Strategic Insight Group CEO Dee Smith why she is auditing companies’ algorithms. O’Neil says it began when she moved from academia to trading during the financial crisis and realized the financial world couldn’t control their algorithms. Now she has turned to corporations more generally, where she believes algorithms have become dangerous “black boxes.” Her view: business leaders need rigorous methodology and tools or they will face catastrophic results that lead to litigation and serious reputational risks. Filmed on June 5, 2019 in New York.

The Effect of Facebook’s Social Media Silo on Itself and You

You reinforce the walls of your personal information bubble. At least, that’s what SUNY Buffalo communication professor Ivan Dyelko and his research team found. In “The dark side of technology: An experimental investigation of the influence of customizability technology on online political selective exposure,” Dyelko and his coauthors report that people are much more likely to click and spend time on articles that reflect their pre-existing biases. In the tests they did, the only group that was likely to spend significant time reading articles that challenged their beliefs was the one in which the news feed was randomized, not weighted by user preference or an algorithm based on prior user behavior. Ask yourself: Do any of your social media services or search engines work on randomization? Or do they all show you what they think you want to see?

So what? Unlike a certain search engine that weights results instead of providing organic returns to user inquiry, social media companies are now shrugging sheepishly and saying, “Yeah, we totally contributed to the siloing of social discourse. Um. Sorry?” (It helps that there’s a study that points out their role in the media failures of 2016.)

.. Who cares? Social media investors should. Facebook is considered a great long-term stock buy right now because it’s virtually monopolizing a market—MySpace is an also-ran, Google+ is a niche product, and LinkedIn was bought so it could become a data collection mechanism for Microsoft’s suite of machine learning-enhanced workplace tools. But Facebook’s monopoly depends on it maintaining its user base of 1.9 billion active users, all of whom generate monetizable content for Facebook. If those users perceive a reason to flee Facebook—if they feel it’s biased, untrustworthy, routinely violating their safety or creating social friction—then the data sets Facebook sells to advertisers and publishers become less valuable, meaning the company eventually loses value.

We’ve already seen this happen with Twitter. Google, Disney and Salesforce all backed away from buying the microblogging company in 2016 for two reasons: Twitter hasn’t been able to successfully monetize its users (unlike Facebook), and Twitter has a growing reputation among current and former users as being unconcerned with the quality of user experience. That experience doesn’t just include the bullying endemic to Twitter; it also includes the content of the tweets themselves. On Twitter, nobody knows if you’re a robot someone paid to promote a specific ideology.

.. And you, dear reader, should care about your filter bubble potential, too. As a recent New York Times op-ed on the virtues of hate-reading reasoned, “Reading what you hate helps you refine what it is you value, whether it’s a style, a story line or an argument… Defensiveness makes you a better reader, a closer, more skeptical reader: a critic. Arguing with the author in your head forces you to gather opposing evidence. You may find yourself turning to other texts with determination, stowing away facts, fighting against the book at hand. You may find yourself developing a point of view.”

If you are the type of person who has actually woken up out of a fitful sleep saying, “And you’re wrong!”—and I’m going to guess, based on the self-selecting nature of people who read So What, Who Cares, every one of you has lost sleep at least once because someone was wrong on the Internet—then you are going to want to be right when you argue on the Internet or in your daily life. Being right requires identifying our own bubble, admitting why you’ve chosen to live in it, and then looking for the points of view that poke at it to see what breaks and what holds true.

What Does Modern Prejudice Look Like?

Harvard psychologist Mahzarin Banaji was once approached by a reporter for an interview. When Banaji heard the name of the magazine the reporter was writing for, she declined the interview: She didn’t think much of the magazine and believed it portrayed research in psychology inaccurately.

 

A patient received better treatment for saying they were a Yale professor than saying they were a quilter.

  • People recognize acts of commission (doing bad stuff to others) as prejudice, not favoritism.

 

In the book, Banaji writes that Kaplan once had a terrible kitchen accident.

.. In many ways, the psychologists’ work mirrors the conclusion of another recent book: In The American Non-Dilemma: Racial Inequality without Racism, sociologist Nancy DiTomaso asks how it is that few people report feeling racial prejudice, while the United States still has enormous disparities. Discrimination today is less about treating people from other groups badly, DiTomaso writes, and more about giving preferential treatment to people who are part of our “in-groups.”

The insidious thing about favoritism is that it doesn’t feel icky in any way, Banaji says. We feel like a great friend when we give a buddy a foot in the door to a job interview at our workplace. We feel like good parents when we arrange a class trip for our daughter’s class to our place of work. We feel like generous people when we give our neighbors extra tickets to a sports game or a show.

In each case, however, Banaji, Greenwald and DiTomaso might argue, we strengthen existing patterns of advantage and disadvantage because our friends, neighbors and children’s classmates are overwhelmingly likely to share our own racial, religious and socioeconomic backgrounds. When we help someone from one of these in-groups, we don’t stop to ask: Whom are we not helping?

 

.. Banaji tells a story in the book about a friend, Carla Kaplan, now a professor at Northeastern University. At the time, both Banaji and Kaplan were faculty members at Yale. Banaji says that Kaplan had a passion — quilting.

“She was washing a big crystal bowl in her kitchen,” Banaji says. “It slipped and it cut her hand quite severely.”

The gash went from Kaplan’s palm to her wrist. She raced over to Yale-New Haven Hospital. Pretty much the first thing she told the ER doctor was that she was a quilter. She was worried about her hand. The doctor reassured her and started to stitch her up. He was doing a perfectly competent job, she says.

But at this moment someone spotted Kaplan. It was a student, who was a volunteer at the hospital.

“The student saw her, recognized her, and said, ‘Professor Kaplan, what are you doing here?’ ” Banaji says.

The ER doctor froze. He looked at Kaplan. He asked the bleeding young woman if she was a Yale faculty member. Kaplan told him she was.

Everything changed in an instant. The hospital tracked down the best-known hand specialist in New England. They brought in a whole team of doctors. They operated for hours and tried to save practically every last nerve.

Banaji says she and Kaplan asked themselves later why the doctor had not called in the specialist right away. “Somehow,” Banaji says, “it must be that the doctor was not moved, did not feel compelled by the quilter story in the same way as he was compelled by a two-word phrase, ‘Yale professor.’ ”

Kaplan told Banaji that she was able to go back to quilting, but that she still occasionally feels a twinge in the hand. And it made her wonder what might have happened if she hadn’t received the best treatment.

Greenwald and Banaji are not suggesting that people stop helping their friends, relatives and neighbors. Rather, they suggest that we direct some effort to people we may not naturally think to help.

After reading the story about Kaplan, for example, one relative of Greenwald’s decided to do something about it. Every year, she used to donate a certain amount of money to her alma mater. After reading Kaplan’s story, Banaji says, the woman decided to keep giving money to her alma mater, but to split the donation in half. She now gives half to her alma mater and half to the United Negro College Fund.

Richard Rohr Meditation: The Source of Action

The effectiveness of action depends on the source from which it springs. If it is coming out of the false self with its shadow side, it is severely limited. If it is coming out of a person who is immersed in God, it is extremely effective. The contemplative state, like the vocation of Our Lady, brings Christ into the world. —Thomas Keating [1]

.. I founded the Center for Action and Contemplation in 1987 because I saw a deep need for the integration of both action and contemplation. Over the years, I met many social activists who were doing excellent social analysis and advocating for crucial justice issues, but they were not working from an energy of love. They were still living out of their false self with the need to win, the need to look good—attached to a superior, politically correct self-image.

They might have the answer, but they are not themselves the answer. In fact, they are often part of the problem. That’s one reason that most revolutions fail and too many reformers self-destruct from within. For that very reason, I believe, Jesus and great spiritual teachers first emphasize transformation of consciousness and soul. Without inner transformation, there is no grounded or lasting reform or revolution. When subjugated people rise to power, they often become as dominating as their oppressors because the same demon of power hasn’t been exorcised in them.

We are easily allured by the next new thing, a new agenda that looks like enlightenment. And then we discover it’s run by unenlightened people who, in fact, love themselves first of all but do not love God or others. They do not really love the Big Truth, but they often love control. Too often, they do not love freedom for everybody but just freedom for their own ideas.

Untransformed liberals often lack the ability to sacrifice the self or create foundations that last. They can’t let go of their own need for change and cannot stand still in a patient, compassionate, and humble way. It is no surprise that Jesus prayed not just for fruit, but “fruit that will last” (John 15:16). Untransformed conservatives, on the other hand, tend to idolize anything that lasts, but then avoid the question, “Is it actually bearing any fruit?” This is the perennial battle between idealism and pragmatism, or romanticism and rationalism.

If we are going to have truly prophetic people who go beyond the categories of liberal and conservative, we have to teach them some way to integrate their needed activism with a truly contemplative mind and heart. I’m convinced that once you learn how to look out at life from the contemplative eyes of the True Self, your politics and economics are going to change on their own. I don’t need to teach you what your politics should or shouldn’t be. Once you see things contemplatively, you’ll begin to seek the bias from the bottom instead of the top, you’ll be free to embrace your shadow, and you can live at peace with those who are different. From a contemplative stance, you’ll know what action is yours to do—and what is not yours to do—almost naturally.