Wrongfully Accused by an Algorithm

In what may be the first known case of its kind, a faulty facial recognition match led to a Michigan man’s arrest for a crime he did not commit.

On a Thursday afternoon in January, Robert Julian-Borchak Williams was in his office at an automotive supply company when he got a call from the Detroit Police Department telling him to come to the station to be arrested. He thought at first that it was a prank.

An hour later, when he pulled into his driveway in a quiet subdivision in Farmington Hills, Mich., a police car pulled up behind, blocking him in. Two officers got out and handcuffed Mr. Williams on his front lawn, in front of his wife and two young daughters, who were distraught. The police wouldn’t say why he was being arrested, only showing him a piece of paper with his photo and the words “felony warrant” and “larceny.”

His wife, Melissa, asked where he was being taken. “Google it,” she recalls an officer replying.

The police drove Mr. Williams to a detention center. He had his mug shot, fingerprints and DNA taken, and was held overnight. Around noon on Friday, two detectives took him to an interrogation room and placed three pieces of paper on the table, face down.

“When’s the last time you went to a Shinola store?” one of the detectives asked, in Mr. Williams’s recollection. Shinola is an upscale boutique that sells watches, bicycles and leather goods in the trendy Midtown neighborhood of Detroit. Mr. Williams said he and his wife had checked it out when the store first opened in 2014.

The detective turned over the first piece of paper. It was a still image from a surveillance video, showing a heavyset man, dressed in black and wearing a red St. Louis Cardinals cap, standing in front of a watch display. Five timepieces, worth $3,800, were shoplifted.

“Is this you?” asked the detective.

The second piece of paper was a close-up. The photo was blurry, but it was clearly not Mr. Williams. He picked up the image and held it next to his face.

“No, this is not me,” Mr. Williams said. “You think all black men look alike?”

Mr. Williams knew that he had not committed the crime in question. What he could not have known, as he sat in the interrogation room, is that his case may be the first known account of an American being wrongfully arrested based on a flawed match from a facial recognition algorithm, according to experts on technology and the law.

A nationwide debate is raging about racism in law enforcement. Across the country, millions are protesting not just the actions of individual officers, but bias in the systems used to surveil communities and identify people for prosecution.

Facial recognition systems have been used by police forces for more than two decades. Recent studies by M.I.T. and the National Institute of Standards and Technology, or NIST, have found that while the technology works relatively well on white men, the results are less accurate for other demographics, in part because of a lack of diversity in the images used to develop the underlying databases.

Last year, during a public hearing about the use of facial recognition in Detroit, an assistant police chief was among those who raised concerns. “On the question of false positives — that is absolutely factual, and it’s well-documented,” James White said. “So that concerns me as an African-American male.”

This monthAmazonMicrosoft and IBM announced they would stop or pause their facial recognition offerings for law enforcement. The gestures were largely symbolic, given that the companies are not big players in the industry. The technology police departments use is supplied by companies that aren’t household names, such as Vigilant Solutions, Cognitec, NEC, Rank One Computing and Clearview AI.

Clare Garvie, a lawyer at Georgetown University’s Center on Privacy and Technology, has written about problems with the government’s use of facial recognition. She argues that low-quality search images — such as a still image from a grainy surveillance video — should be banned, and that the systems currently in use should be tested rigorously for accuracy and bias.

“There are mediocre algorithms and there are good ones, and law enforcement should only buy the good ones,” Ms. Garvie said.

About Mr. Williams’s experience in Michigan, she added: “I strongly suspect this is not the first case to misidentify someone to arrest them for a crime they didn’t commit. This is just the first time we know about it.”

Mr. Williams’s case combines flawed technology with poor police work, illustrating how facial recognition can go awry.

The Shinola shoplifting occurred in October 2018. Katherine Johnston, an investigator at Mackinac Partners, a loss prevention firm, reviewed the store’s surveillance video and sent a copy to the Detroit police, according to their report.

Five months later, in March 2019, Jennifer Coulson, a digital image examiner for the Michigan State Police, uploaded a “probe image” — a still from the video, showing the man in the Cardinals cap — to the state’s facial recognition database. The system would have mapped the man’s face and searched for similar ones in a collection of 49 million photos.

The state’s technology is supplied for $5.5 million by a company called DataWorks Plus. Founded in South Carolina in 2000, the company first offered mug shot management software, said Todd Pastorini, a general manager. In 2005, the firm began to expand the product, adding face recognition tools developed by outside vendors.

When one of these subcontractors develops an algorithm for recognizing faces, DataWorks attempts to judge its effectiveness by running searches using low-quality images of individuals it knows are present in a system. “We’ve tested a lot of garbage out there,” Mr. Pastorini said. These checks, he added, are not “scientific” — DataWorks does not formally measure the systems’ accuracy or bias.

“We’ve become a pseudo-expert in the technology,” Mr. Pastorini said.

In Michigan, the DataWorks software used by the state police incorporates components developed by the Japanese tech giant NEC and by Rank One Computing, based in Colorado, according to Mr. Pastorini and a state police spokeswoman. In 2019, algorithms from both companies were included in a federal study of over 100 facial recognition systems that found they were biased, falsely identifying African-American and Asian faces 10 times to 100 times more than Caucasian faces.

Rank One’s chief executive, Brendan Klare, said the company had developed a new algorithm for NIST to review that “tightens the differences in accuracy between different demographic cohorts.”

After Ms. Coulson, of the state police, ran her search of the probe image, the system would have provided a row of results generated by NEC and a row from Rank One, along with confidence scores. Mr. Williams’s driver’s license photo was among the matches. Ms. Coulson sent it to the Detroit police as an “Investigative Lead Report.”

This document is not a positive identification,” the file says in bold capital letters at the top. “It is an investigative lead only and is not probable cause for arrest.

This is what technology providers and law enforcement always emphasize when defending facial recognition: It is only supposed to be a clue in the case, not a smoking gun. Before arresting Mr. Williams, investigators might have sought other evidence that he committed the theft, such as eyewitness testimony, location data from his phone or proof that he owned the clothing that the suspect was wearing.

In this case, however, according to the Detroit police report, investigators simply included Mr. Williams’s picture in a “6-pack photo lineup” they created and showed to Ms. Johnston, Shinola’s loss-prevention contractor, and she identified him. (Ms. Johnston declined to comment.)

Mr. Pastorini was taken aback when the process was described to him. “It sounds thin all the way around,” he said.

Mr. Klare, of Rank One, found fault with Ms. Johnston’s role in the process. “I am not sure if this qualifies them as an eyewitness, or gives their experience any more weight than other persons who may have viewed that same video after the fact,” he said. John Wise, a spokesman for NEC, said: “A match using facial recognition alone is not a means for positive identification.”

The Friday that Mr. Williams sat in a Detroit police interrogation room was the day before his 42nd birthday. That morning, his wife emailed his boss to say he would miss work because of a family emergency; it broke his four-year record of perfect attendance.

In Mr. Williams’s recollection, after he held the surveillance video still next to his face, the two detectives leaned back in their chairs and looked at one another. One detective, seeming chagrined, said to his partner: “I guess the computer got it wrong.”

They turned over a third piece of paper, which was another photo of the man from the Shinola store next to Mr. Williams’s driver’s license. Mr. Williams again pointed out that they were not the same person.

Mr. Williams asked if he was free to go. “Unfortunately not,” one detective said.

Mr. Williams was kept in custody until that evening, 30 hours after being arrested, and released on a $1,000 personal bond. He waited outside in the rain for 30 minutes until his wife could pick him up. When he got home at 10 p.m., his five-year-old daughter was still awake. She said she was waiting for him because he had said, while being arrested, that he’d be right back.

She has since taken to playing “cops and robbers” and accuses her father of stealing things, insisting on “locking him up” in the living room.

The Williams family contacted defense attorneys, most of whom, they said, assumed Mr. Williams was guilty of the crime and quoted prices of around $7,000 to represent him. Ms. Williams, a real estate marketing director and food blogger, also tweeted at the American Civil Liberties Union of Michigan, which took an immediate interest.

We’ve been active in trying to sound the alarm bells around facial recognition, both as a threat to privacy when it works and a racist threat to everyone when it doesn’t,” said Phil Mayor, an attorney at the organization. “We know these stories are out there, but they’re hard to hear about because people don’t usually realize they’ve been the victim of a bad facial recognition search.”

Two weeks after his arrest, Mr. Williams took a vacation day to appear in a Wayne County court for an arraignment. When the case was called, the prosecutor moved to dismiss, but “without prejudice,” meaning Mr. Williams could later be charged again.

Maria Miller, a spokeswoman for the prosecutor, said a second witness had been at the store in 2018 when the shoplifting occurred, but had not been asked to look at a photo lineup. If the individual makes an identification in the future, she said, the office will decide whether to issue charges.

A Detroit police spokeswoman, Nicole Kirkwood, said that for now, the department “accepted the prosecutor’s decision to dismiss the case.” She also said that the department updated its facial recognition policy in July 2019 so that it is only used to investigate violent crimes.

The department, she said in another statement, “does not make arrests based solely on facial recognition. The investigator reviewed video, interviewed witnesses, conducted a photo lineup.”

On Wednesday, the A.C.L.U. of Michigan filed a complaint with the city, asking for an absolute dismissal of the case, an apology and the removal of Mr. Williams’s information from Detroit’s criminal databases.

The Detroit Police Department “should stop using facial recognition technology as an investigatory tool,” Mr. Mayor wrote in the complaint, adding, “as the facts of Mr. Williams’s case prove both that the technology is flawed and that DPD investigators are not competent in making use of such technology.”

Mr. Williams’s lawyer, Victoria Burton-Harris, said that her client is “lucky,” despite what he went through.

He is alive,” Ms. Burton-Harris said. He is a very large man. My experience has been, as a defense attorney, when officers interact with very large men, very large black men, they immediately act out of fear. They don’t know how to de-escalate a situation.”

Mr. Williams and his wife have not talked to their neighbors about what happened. They wonder whether they need to put their daughters into therapy. Mr. Williams’s boss advised him not to tell anyone at work.

My mother doesn’t know about it. It’s not something I’m proud of,” Mr. Williams said. “It’s humiliating.”

He has since figured out what he was doing the evening the shoplifting occurred. He was driving home from work, and had posted a video to his private Instagram because a song he loved came on — 1983’s “We Are One,” by Maze and Frankie Beverly. The lyrics go:

I can’t understand

Why we treat each other in this way

Taking up time

With the silly silly games we play

He had an alibi, had the Detroit police checked for one.

Aaron Krolik contributed reporting.

Facial Recognition Tech Is Growing Stronger, Thanks to Your Face

One database, which dates to 2014, was put together by researchers at Stanford. It was called Brainwash, after a San Francisco cafe of the same name, where the researchers tapped into a camera. Over three days, the camera took more than 10,000 images, which went into the database, the researchers wrote in a 2015 paper. The paper did not address whether cafe patrons knew their images were being taken and used for research. (The cafe has closed.)

The Stanford researchers then shared Brainwash. According to research papers, it was used in China by academics associated with the National University of Defense Technology and Megvii, an artificial intelligence company that The New York Times previously reported has provided surveillance technology for monitoring Uighurs.

The Brainwash data set was removed from its original website last month after Adam Harvey, an activist in Germany who tracks the use of these repositories through a website called MegaPixels, drew attention to it. Links between Brainwash and papers describing work to build A.I. systems at the National University of Defense Technology in China have also been deleted, according to documentation from Mr. Harvey.

Stanford researchers who oversaw Brainwash did not respond to requests for comment. “As part of the research process, Stanford routinely makes research documentation and supporting materials available publicly,” a university official said. “Once research materials are made public, the university does not track their use nor did university officials.”

Duke University researchers also started a database in 2014 using eight cameras on campus to collect images, according to a 2016 paper published as part of the European Conference on Computer Vision. The cameras were denoted with signs, said Carlo Tomasi, the Duke computer science professor who helped create the database. The signs gave a number or email for people to opt out.

The Duke researchers ultimately gathered more than two million video frames with images of over 2,700 people, according to the paper. They also posted the data set, named Duke MTMC, online. It was later cited in myriad documents describing work to train A.I. in the United States, in China, in Japan, in Britain and elsewhere.

.. An OkCupid spokeswoman said Clarifai contacted the company in 2014 “about collaborating to determine if they could build unbiased A.I. and facial recognition technology” and that the dating site “did not enter into any commercial agreement then and have no relationship with them now.” She did not address whether Clarifai had gained access to OkCupid’s photos without its consent.

Clarifai used the images from OkCupid to build a service that could identify the age, sex and race of detected faces, Mr. Zeiler said. The start-up also began working on a tool to collect images from a website called Insecam — short for “insecure camera” — which taps into surveillance cameras in city centers and private spaces without authorization. Clarifai’s project was shut down last year after some employees protested and before any images were gathered, he said.

The Ethical Dilemma Facing Silicon Valley’s Next Generation

Stanford has established itself as the epicenter of computer science, and a farm system for the tech giants. Following major scandals at Facebook, Google, and others, how is the university coming to grips with a world in which many of its students’ dream jobs are now vilified?

At Stanford University’s business school, above the stage where Elizabeth Holmes once regurgitated the myths of Silicon Valley, there now hangs a whistle splattered in blood. More than 500 people have gathered to hear the true story of Theranos, the $9 billion blood-testing company Holmes launched in 2004 as a Stanford dropout with the help of one of the school’s famed chemical engineering professors.

When Holmes was weaving the elaborate lies that ultimately led to the dissolution of her company, she leaned heavily on tech truisms that treat dogged pursuit of market domination as a virtue. “The minute that you have a backup plan, you’ve admitted that you’re not going to succeed,” she said onstage in 2015. But Shultz and Cheung, who faced legal threats from Theranos for speaking out, push back against the idea of pursuing a high-minded vision at all costs. “We don’t know how to handle new technologies anymore,” Cheung says, “and we don’t know the consequences necessarily that they’ll have.”

The words resonate in the jam-packed auditorium, where students line up afterward to nab selfies with and autographs from the whistleblowers. Kendall Costello, a junior at Stanford, idolized Holmes in high school and imagined working for Theranos one day. Now she’s more interested in learning how to regulate tech than building the next product that promises to change the world. “I really aspired to kind of be like her in a sense,” Costello says. “Then two years later, in seeing her whole empire crumble around her, in addition to other scandals like Facebook’s Cambridge Analytica and all these things that are coming forward, I was just kind of disillusioned.”

..But the endless barrage of negative news in tech, ranging from Facebook fueling propaganda campaigns by Russian trolls to Amazon selling surveillance software to governments, has forced Stanford to reevaluate its role in shaping the Valley’s future leaders. Students are reconsidering whether working at Google or Facebook is landing a dream job or selling out to craven corporate interests. Professors are revamping courses to address the ethical challenges tech companies are grappling with right now. And university president Marc Tessier-Lavigne has made educating students on the societal impacts of technology a tentpole of his long-term strategic plan.

As tech comes to dominate an ever-expanding portion of our daily lives, Stanford’s role as an educator of the industry’s engineers and a financier of its startups grows increasingly important. The school may not be responsible for creating our digital world, but it trains the architects. And right now, students are weighing tough decisions about how they plan to make a living in a world that was clearly constructed the wrong way. “To me it seemed super empowering that a line of code that I wrote could be used by millions of people the next day,” says Matthew Sun, a junior majoring in computer science and public policy, who helped organize the Theranos event. “Now we’re realizing that’s maybe not always a good thing.”

.. Because membership costs $21,000 per year, the career fairs tend to attract only the most renowned firms.

Honestly, I think they’re horrific,” says Vicki Niu, a 2018 Stanford graduate who majored in computer science. She recalls her first career fair being as hectic as a Black Friday sale, with the put-on exclusivity of a night club. (Students must present their Stanford IDs to enter the tent.) But like other freshmen, she found herself swept up in the pursuit of an internship at a large, prestigious tech firm. “Everybody is trying to get interviews at Google and Facebook and Palantir,” she says. “There’s all this hype around them. Part of my mind-set coming in was that I wanted to learn, but I think there was definitely also this big social pressure and this desire to prove yourself and to prove to people that you’re smart.”

Stanford’s computer science department has long been revered for its graduate programs—Google was famously built as a research project by Ph.D. students Larry Page and Sergey Brin—but the intense interest among undergrads is relatively new. In 2007, the school conferred more bachelor’s degrees in English (92) than computer science (70). The next year, though, Stanford revamped its CS curriculum from a one-size-fits-all education to a more flexible framework that funneled students along specialized tracks such as graphics, human-computer interaction, and artificial intelligence. “We needed to make the major more attractive, to show that computer science isn’t just sitting in a cube all day,” Mehran Sahami, a computer science professor who once worked at Google, said later.

The change in curriculum coincided with an explosion of wealth and perceived self-importance in the Valley. The iPhone opened up the potential for thousands of new businesses built around apps, and when its creator died he earned rapturous comparisons to Thomas Edison. Facebook emerged as the fastest-growing internet company of all time, and the Arab Spring made its influence seem benign rather than ominous. As the economy recovered from the recession, investors decided to park their money in startups like Uber and Airbnb that might one day become the next Google or Amazon. A 2013 video by the nonprofit Code.org featured CEOs, Chris Bosh, and will.i.am comparing computer programmers to wizards, superheroes, and rock stars.

Stanford and its students eagerly embraced this cultural shift. John Hennessy, a computer science professor who became president of the university from 2000 to 2016, served on Google’s board of directors and is now the executive chairman of Google parent company Alphabet. LinkedIn founder and Stanford alum Reid Hoffman introduced a new computer science course called Blitzscaling and brought in high-profile entrepreneurs to teach students how to “build massive organizations, user bases, and businesses, and to do so at a dizzyingly rapid pace.” (Elizabeth Holmes was among the speakers.) Mark Zuckerberg became an annual guest in Sahami’s popular introductory computer science class. “It just continued to emphasize how privileged Stanford students are in so many ways, that we have the CEO of Facebook taking time out of his day to come talk to us,” says Vinamrata Singal, a 2016 graduate who had Zuckerberg visit her class freshman year. “It felt really surreal and it did make me excited to want to continue studying computer science.”

In 2013, Stanford began directly investing in students’ companies, much like a venture capital firm. Even without direct Stanford funding, the school’s proximity to wealth helped plenty of big ideas get off the ground. Evan Spiegel, who was a junior at Stanford in 2011 when he started working on Snapchat, connected with his first investor via a Stanford alumni network on Facebook. “Instead of starting a band or trying to make an independent movie or blogging, people would get into code,” says Billy Gallagher, a 2014 graduate who was the editor-in-chief of the school newspaper. “It was a similar idea to, ‘Here’s our band’s vinyl or our band’s tape. Come see us play.’”

..But it’s not just that coding was a creative outlet, as is often depicted in tech origin stories. Working at a big Silicon Valley company also became a path to a specific kind of upper-crust success that students at top schools are groomed for. “Why do so many really bright young kids go into consulting and banking?” asks Gallagher. “They’re prestigious so your parents can be proud of you, they pay really well, and they put you on a career path to open up new doors. Now we’re seeing that’s happening a lot with Google and Facebook.”

By the time Niu arrived in 2014, computer science had become the most popular major on campus and 90 percent of undergrads were taking at least one CS course. As a high schooler, her knowledge of Silicon Valley didn’t extend much further than The Internship, a Vince Vaughn–Owen Wilson comedy about working at Google that doubled as a promotional tool for the search giant. She soon came to realize that landing a job at one of the revered tech giants or striking it rich with an app were Stanford’s primary markers of success. Her coursework was largely technical, focusing on the process of coding and not so much on the outcomes. And in the rare instances when Niu heard ethics discussed in class, it was often framed around the concerns of tech’s super-elite, like killer robots destroying humanity in the future. “In my computer science classes and just talking to other people who were interested in technology, it didn’t seem like anybody really cared about social impact,” she says. “Or if they did, they weren’t talking about it.”

In the spring of her freshman year, Niu and two other students hosted a meeting to gauge interest in a new group focused on socially beneficial uses of technology. The computer science department provided funding for red curry and pad thai. Niu was shocked when the food ran out, as more than 100 students showed up for the event. “Everybody had the same experience: ‘I’m a computer science student. I’m doing this because I want to create an impact. I feel like I’m alone.’”

From this meeting sprang the student organization CS + Social Good. It aimed to expose students to professional opportunities that existed outside the tech giants and the hyperaggressive startups that aspired to their stature. In its first year, the group developed new courses about social-impact work, brought in speakers to discuss positive uses of technology, and offered summer fellowships to get students interning at nonprofits instead of Apple or Google. Hundreds of students and faculty engaged with the organization’s programming.

In Niu’s mind, “social good” referred mainly to the positive applications of technology. But stopping bad uses of tech is just as important as promoting good ones. That’s a lesson the entire Valley has been forced to reckon with as its benevolent reputation has unraveled. “Most of our programming had been, ‘Look at these great ways you can use technology to help kids learn math,’” Niu says. “There was this real need to not only talk about that, but to also be like, ‘It’s not just that technology is neutral. It can actually be really harmful.’”

Many students find it difficult to pinpoint a specific transgression that flipped their perception of Silicon Valley, simply because there have been so many.

The torrid pace of bad news has been jarring for students who entered school with optimistic views of tech. Nichelle Hall, a senior majoring in computer science, viewed Google as the ideal landing spot for an aspiring software engineer when she started college. “I associated it so much with success,” she says. “It’s the first thing I thought about when I thought about technology.” But when she was offered an on-site interview for a potential job at the search giant in the fall, she declined. Project Dragonfly, Google’s (reportedly abandoned) effort to bring a censored search engine to China, gave her pause. It wasn’t just that she objected to the idea on principle. She sensed that working for such a large corporation would likely put her personal morals and corporate directives in conflict. “They say don’t do evil and then they do things like that,” she says. “I wasn’t really into the big-company idea for that reason. … You don’t necessarily know what the intentions of your executives are.”

  • ..Google has hardly been the most damaged brand during the techlash. (The company says it has not seen a year-over-year decline in Stanford recruits to this point.)
  • Students repeatedly bring up Facebook as a company that’s fallen out of favor.
  • Uber, with its cascade of controversies, now has to “fight to try and get people in,” according to junior Jose Giron.
  • And Palantir, the secretive data-mining company started by Stanford alum Peter Thiel, has also lost traction due to Thiel’s ties to Trump and worries that the company could help the president develop tech to advance his draconian immigration policies. “There’s a growing concern over your personal decision where to work after graduation,” Sun says.

There’s a lot of personal guilt around pursuing CS. If you do that, people call you a sellout or you might view yourself as a sellout. If you take a high-paying job, people might say, ‘Oh, you’re just going to work for a big tech company. All you care about is yourself.’”

Landing a job at a major tech firm is often as much about prestige as passion, which is one reason the CS major has expanded so dramatically. But a company’s tarnished reputation can transfer to its employees. Students debate whether fewer of their peers are actually taking gigs at Facebook, or whether they’re just less vocal in bragging about it. At lunch at a Burmese restaurant on campus, Hall and Sun summed up the transition succinctly. “No one’s like, ‘I got an internship at Uber!’” Sun says. Hall follows up: “They’re like, ‘I got an internship … at Uber …’”

The concerns are bigger than which companies rise or fall in the estimation of up-and-coming engineers. Stanford and computer science programs across the country may not be adequately equipped to wade through the ethical minefield that is expanding along with tech’s influence. Sahami acknowledges that many computer science classes are designed to teach students how to solve technical problems rather than to think about the real-world issues that a solution might create. Part of the challenge comes from computer science being a young discipline compared to other engineering fields, meaning that practical examples of malpractice are emerging in real time from today’s headlines.

Vik Pattabi, a senior majoring in computer science, originally studied mechanical engineering. In those classes, students are constantly reminded of the 1940 collapse of Tacoma Narrows Bridge: A modern marvel was destroyed because its highly educated engineers did not foresee all the possible threats to their creation (in that case, the wind). Pattabi’s CS coursework hasn’t yet included a comparable example. “A lot of the second- and third-order effects that we see [in] Silicon Valley have happened in the last two or three years,” Pattabi says. “The department is trying to react as fast as it can, but they don’t have 30 years of case studies to work with.”

Another issue is the longstanding divide on campus between the engineering types—known as “techies”—and the humanities or social sciences majors, known as “fuzzies.” Though the school has focused more on interdisciplinary studies in recent years, there remains a gap in understanding that’s often filled in by stereotype. This sort of divide is a common aspect of college life, but the stakes feel higher when some of the students will one day be programming the algorithms that govern the digital world. “There’s things [said] like, ‘You can’t spell fascist without CS.’ People will tell you things like that,” Hall says. “I think people may feel antagonized.”

The school’s deep ties to the Bay Area’s corporate giants, long a much-touted recruitment tool, suddenly look different in light of the problems that the industry has created. At the January career fair, members of Students for the Liberation of All Peoples (SLAP), an activist group on campus that aims to disrupt Stanford’s “culture of apathy,” handed out flyers that urged students not to work at Amazon and Salesforce because of their commercial ties to ICE and the United States Border Patrol. (Employees at the companies have raised similar concerns.) “REFUSE to be part of the Stanford → racist tech pipeline,” the flyer reads, in part.

Two students in the group said they were asked to leave the career fair by Computer Forum officials. When the students refused to comply, they say they were escorted out by campus police under threat of arrest for disrupting a private event. A Stanford spokesperson confirmed the incident. “The protesting students were disruptive and asked by police to leave,” the spokesperson said in an emailed statement. “The students were given the option to protest outside the event or in White Plaza. They chose to leave.”

For members of SLAP, the exchange reinforced the ways in which Stanford institutionally and culturally cuts itself off from the issues occurring in the real world. “You might hear this idea of the ‘Stanford bubble,’ where Stanford students kind of just stay on campus and they just do what they need to do for their classes and their jobs,” says Kimiko Hirota, a SLAP member and junior majoring in sociology and comparative studies in race and ethnicity, who participated in the career fair protest. She said many of the students she talked to had no idea about the tech firms’ government contracts. “To me the amount of students on campus that are politically engaged and are actively using their Stanford privilege for a greater good is extremely small.”

The computer science major includes a “technology in society” course requirement that can be fulfilled by a number of ethics classes, and teaching students about their ethical responsibilities is a component of the department’s accreditation process. CS + Social Good has expanded its footprint on campus, teaching more classes and organizing more events like the Theranos talk starring the whistleblowers. Yet the flexibility of the CS major cuts both ways. It means that students who care to take a holistic approach to the discipline can combine rigorous training in code with an education in ethics; it also means that it remains all too easy for some students to avoid engaging with the practical ramifications of their work. “You can very much come to Stanford feeling very apathetic about the impact of the technology and leave just that way without any effort,” Hall says. “I don’t feel as though we are forced to encounter the impact.”

On a Wednesday afternoon, students spill into a lobby in front of a standing-room-only auditorium in the School of Education, where Jeremy Weinstein is talking about the promise and perils of using algorithms in criminal justice. Next year Californians will vote on a bill that would replace cash bail with a computerized risk-assessment system that calculates an arrested person’s likelihood of returning for a court appearance. The idea is to give people who can’t afford to make bail another way to get out of jail through a fairer policy. But such algorithms have been found to reinforce racial biases in the criminal justice system, according to a ProPublicainvestigation. Instead of being a solution to an unfair process, poorly implemented software could create an entirely new form of systemic discrimination. Students were asked to vote on whether they supported cash bail or the algorithm. The class was evenly split. Unlike in most CS classes, Weinstein could not offer students the comfort of a “correct” answer. “We need to deconstruct these algorithms in order to help people see that technology is not just something to be trusted,” he says. “It’s not just something that’s objective and fair because it’s numerical, but it actually reflects a set of choices that people make.”

Though Weinstein is a political science professor, he’s one of three educators leading the new version of the CS department’s flagship ethics course, CS181. Teaching with him are Sahami, the computer science professor, and Rob Reich, a political science professor and philosopher. The trio devised the course structure over a series of coffee-fueled meetings as the tech backlash unfolded during the past year and a half. After discussing algorithmic bias, the class will explore privacy in the age of facial recognition, the social impacts of autonomous technology, and the responsibilities of private platforms in regard to free speech. The coursework is meant to be hands-on. During the current unit, students must build their own risk-assessment algorithm using an actual criminal history data set, then assess it for fairness. “We run it like a talk show,” Weinstein says. “There’s a lot of call-and-response, asking questions, getting people to talk in small groups.”

While Stanford’s computer science program has had an ethics component for decades, this course marks the first time that experts in other fields are so directly involved in the curriculum. About 300 students have enrolled in the course, including majors in history, philosophy, and biology. It provides an opportunity for the techies and fuzzies to learn from one another, and for professors removed from the Valley’s tech culture to contextualize the industry’s societal impacts. In the course overview materials, the moral reckoning occurring in the tech sector today is compared to the advent of the nuclear bomb.

The course’s popularity is a sign that the gravity of the moment is weighing on many Stanford minds. Antigone Xenopoulos, a junior majoring in symbolic systems (a techie-fuzzie hybrid major that incorporates computer science, linguistics, and philosophy), is a research assistant for CS181. She wasn’t the only student who quoted a line from Spider-Man to me—with great power comes great responsibilitywhen referencing the current landscape. “If they’re going to give students the tools to have such immense influence and capabilities, [Stanford] should also guide those students in developing ethical compasses,” she says.

 ..While the early years of the decade saw prominent tech executives like Holmes and Zuckerberg teaching students how to lifehack their way to success, the new ethics course will bring in guest speakers from WhatsApp, Facebook, and the NSA to answer “hard questions,” Sahami says “I wouldn’t say industry is influencing Stanford,” he says. “I would say the relationship with industry allows us to have more authentic conversations where we’re really bringing in people who are decision-makers in these areas.”
.. Some of the more critical voices from within the industry are also taking on more permanent roles at Stanford. Alex Stamos, the former chief security officer at Facebook, taught a “hack lab” for non-CS majors last fall, helping them understand cybersecurity threats. He’s now developing a more advanced computer science course, to be piloted later this year, that explores trust and safety issues in the era of misinformation and widespread online harassment. Stamos led Facebook’s internal investigation into Russian political interference on the platform and clashed with top executives over how much of that information should be made public. He left the company in August to join Stanford, where he hopes to impart lessons from his time battling a digital attack that was waged not through hacking, but through ad purchases, incendiary memes, and politically charged Facebook events. “One of the things we don’t teach computer science students is all of the non-technically advanced abuses of technology that cause real harm,” Stamos says. “I want to expose students to [things like], ‘These are the mistakes that were made before, these are the kinds of problems that existed, and these are the company’s reactions to those mistakes.’”

Stamos rejects the idea that ethics is the correct framework to think about addressing tech’s most pressing issues. “The problem here is not that people are making decisions that are straight-up evil,” he says. “The problem is that people are not foreseeing the outcomes of their actions. Part of that is a lack of paranoia. One of our problems in Silicon Valley is we build products to be used the right way. … It’s hard to envision all the misuses unless you understand all the things that have come before.”

While he says that Stanford bears some responsibility for the Valley’s tunnel vision, he praises the school for welcoming tech leaders with recent, relevant experiences to help students prepare for emerging threats. “When I was going to school, computers were important, but we weren’t talking about building companies that might change history,” Stamos says. “The students who come to me are really interested in the impact of what they do on society.”

Stamos regularly fields questions from students about whether to work at Facebook or Google. He tells them that they should, not in spite of the companies’ mounting issues, but because of them.If you actually care about making communication technologies compatible with democracy, then the place to be is at one of the companies that actually has the problems,” he says. “Not working at the big places that could actually solve it does not make things better.”

The tech giants continue to consolidate power even as they face withering criticism. Facebook’s user base growth accelerated last quarter despite its scandals. Uber will go public this year at a valuation as high as $120 billion. Apple, Amazon, and Google are all planning to open large new offices around the country in the near future. And for all the optimistic talk of working at ethically minded startups among students, creation of nascent businesses is at roughly a 40-year low in the United States. Small firms that enter the terrain of the Frightful Five are typically acquired or destroyed.

It is hard to find a Stanford computer science student, even among the ethically minded set CS + Social Good has helped cultivate, who will publicly proclaim that they’ll never work for one of these dominant companies, as all of them offer opportunities for high pay, engaging individual work, and comforting job security. International students have to worry about securing work visas however they can; students on financial aid may need to make enough money to support other family members. And for many others, it’s not clear that anything that’s happened in the Valley is truly beyond the pale. In that sense, the engineers are just like us, aghast at the headlines but still clicking away inside a system that’s come to feel inescapable. “These events feel too big for most students to take into account,” says Jason He, a master’s student in electrical engineering. “At the end of the day, I think for a lot of students who have been paying a lot of money for their education, if the six-figure salary is offered, it’s pretty hard for students to turn down.”

There is still an opportunity, the thinking at Stanford goes, for every company to do good. Nichelle Hall, the senior who declined the Google interview, landed a job working on Medium’s trust and safety team. But she recognizes that she may have set her qualms aside if Google had been her only employment option. “Some of the feedback that CS + Social Good gets is, ‘Oh, the members end up working for Facebook, they end up working for Google,’” says Hall, who’s been involved with the organization since 2017. “People who care about this intersection of social impact and computer science will go to these companies and do a better job than if they weren’t interested in this stuff.”

Impact is the word that I heard more than any other while on campus. It’s how students framed their decision to go to Stanford, to pursue a career in computer science, to do good in the world after graduating. It’s a word that Hoffman used to describe his Blitzscaling class, and one Holmes used to explain to students why she dropped out of school. “I had the tools that I needed to be able to go out and begin making this impact,” she said. It’s the currency of Silicon Valley, that people spend for good and for ill.

The ability to create impact with a few lines of code has long been what separated software engineers from the rest of us, and turned the Valley into a self-proclaimed utopia of young rebels using technology to save the world from its older, antiquated self. But that’s not the image anymore. Now aspiring engineers draw a comparison between their chosen profession and investment banking. The finance industry wrecked the world a decade ago because of its misunderstanding of complex, automated systems that spun out of control—and its confidence that someone else would ultimately pay the price if things went wrong. You see this confidence in Zuckerberg’s incredulous response when anyone suggests that he resign, and in Google CEO Sundar Pichai’s initial refusal to testify before Congress. And you can see it at Stanford, where the endowment has never been higher, the fundraising has never been easier, and the career fair is still filled with slogans vowing to make the world a better place.

Perhaps this entire strip of land known as the Valley will fully calcify into a West Coast Wall Street, where the people with all the insider knowledge profit off the muppets who can’t stop using their products. If today’s young tech skeptics turn to cynics when they enter the working world, such a future is easy to imagine. But—and this is the hopeful, intoxicating, dangerous thing about technology—there’s always bright minds out there who think they can build a solution that just might fix this mess we’ve made. And people, especially young people, will always be enthralled by the romance of a new idea. “We’re creating things that haven’t necessarily existed before, and so we won’t be able to anticipate all the challenges that we have,” says Hall, who graduates in just four months. “But once we do, it’s important that we can reconcile them with grace and humility. I’m sure it will be a hard job, but it’s important that it’s hard. I’m up for the challenge.”

Warning! Everything Is Going Deep: ‘The Age of Surveillance Capitalism’

Deep learning, deep insights, deep artificial minds — the list goes on and on. But with unprecedented promise comes some unprecedented peril.

Around the end of each year major dictionaries declare their “word of the year.” Last year, for instance, the most looked-up word at Merriam-Webster.com was “justice.” Well, even though it’s early, I’m ready to declare the word of the year for 2019.

The word is “deep.”

Why? Because recent advances in the speed and scope of digitization, connectivity, big data and artificial intelligence are now taking us “deep” into places and into powers that we’ve never experienced before — and that governments have never had to regulate before. I’m talking about

  • deep learning,
  • deep insights,
  • deep surveillance,
  • deep facial recognition,
  • deep voice recognition,
  • deep automation and
  • deep artificial minds.

..Which is why it may not be an accident that one of the biggest hit songs today is “Shallow,” from the movie “A Star Is Born.” The main refrain, sung by Lady Gaga and Bradley Cooper, is: “I’m off the deep end, watch as I dive in. … We’re far from the shallow now.”

.. We sure are. But the lifeguard is still on the beach and — here’s what’s really scary — he doesn’t know how to swim! More about that later. For now, how did we get so deep down where the sharks live?

The short answer: Technology moves up in steps, and each step, each new platform, is usually biased toward a new set of capabilities. Around the year 2000 we took a huge step up that was biased toward connectivity, because of the explosion of fiber-optic cable, wireless and satellites.

Suddenly connectivity became so fast, cheap, easy for you and ubiquitous that it felt like you could touch someone whom you could never touch before and that you could be touched by someone who could never touch you before.

Around 2007, we took another big step up. The iPhone, sensors, digitization, big data, the internet of things, artificial intelligence and cloud computing melded together and created a new platform that was biased toward abstracting complexity at a speed, scope and scale we’d never experienced before.

So many complex things became simplified. Complexity became so fast, free, easy to use and invisible that soon with one touch on Uber’s app you could page a taxi, direct a taxi, pay a taxi, rate a taxi driver and be rated by a taxi driver.