As regulators seek ways to curb the company’s power, there is more focus on the vast index — hundreds of billions of web pages — behind its search engine.
In 2000, just two years after it was founded, Google reached a milestone that would lay the foundation for its dominance over the next 20 years: It became the world’s largest search engine, with an index of more than one billion web pages.
The rest of the internet never caught up, and Google’s index just kept on getting bigger. Today, it’s somewhere between 500 billion and 600 billion web pages, according to estimates.
Now, as regulators around the world examine ways to curb Google’s power, including a search monopoly case expected from state attorneys general as early as this week and the antitrust lawsuit the Justice Department filed in October, they are wrestling with a company whose sheer size has allowed it to squash competitors. And those competitors are pointing investigators toward that enormous index, the gravitational center of the company.
“If people are on a search engine with a smaller index, they’re not always going to get the results they want. And then they go to Google and stay at Google,” said Matt Wells, who started Gigablast, a search engine with an index of around five billion web pages, about 20 years ago. “A little guy like me can’t compete.”
Understanding how Google’s search works is a key to figuring out why so many companies find it nearly impossible to compete and, in fact, go out of their way to cater to its needs.
Every search request provides Google with more data to make its search algorithm smarter. Google has performed so many more searches than any other search engine that it has established a huge advantage over rivals in understanding what consumers are looking for. That lead only continues to widen, since Google has a market share of about 90 percent.
Google directs billions of users to locations across the internet, and websites, hungry for that traffic, create a different set of rules for the company. Websites often provide greater and more frequent access to Google’s so-called web crawlers — computers that automatically scour the internet and scan web pages — allowing the company to offer a more extensive and up-to-date index of what is available on the internet.
When he was working at the music site Bandcamp, Zack Maril, a software engineer, became concerned about how Google’s dominance had made it so essential to websites.
In 2018, when Google said its crawler, Googlebot, was having trouble with one of Bandcamp’s pages, Mr. Maril made fixing the problem a priority because Google was critical to the site’s traffic. When other crawlers encountered problems, Bandcamp would usually block them.
Mr. Maril continued to research the different ways that websites opened doors for Google and closed them for others. Last year, he sent a 20-page report, “Understanding Google,” to a House antitrust subcommittee and then met with investigators to explain why other companies could not recreate Google’s index.
“It’s largely an unchecked source of power for its monopoly,” said Mr. Maril, 29, who works at another technology company that does not compete directly with Google. He asked that The New York Times not identify his employer since he was not speaking for it.
A report this year by the House subcommittee cited Mr. Maril’s research on Google’s efforts to create a real-time map of the internet and how this had “locked in its dominance.” While the Justice Department is looking to unwind Google’s business deals that put its search engine front and center on billions of smartphones and computers, Mr. Maril is urging the government to intervene and regulate Google’s index. A Google spokeswoman declined to comment.
Websites and search engines are symbiotic. Websites rely on search engines for traffic, while search engines need access to crawl the sites to provide relevant results for users. But each crawler puts a strain on a website’s resources in server and bandwidth costs, and some aggressive crawlers resemble security risks that can take down a site.
Since having their pages crawled costs money, websites have an incentive to let it be done only by search engines that direct enough traffic to them. In the current world of search, that leaves Google and — in some cases — Microsoft’s Bing.
Google and Microsoft are the only search engines that spend hundreds of millions of dollars annually to maintain a real-time map of the English-language internet. That’s in addition to the billions they’ve spent over the years to build out their indexes, according to a report this summer from Britain’s Competition and Markets Authority.
Google holds a significant leg up on Microsoft in more than market share. British competition authorities said Google’s index included about 500 billion to 600 billion web pages, compared with 100 billion to 200 billion for Microsoft.
Other large tech companies deploy crawlers for other purposes. Facebook has a crawler for links that appear on its site or services. Amazon says its crawler helps improve its voice-based assistant, Alexa. Apple has its own crawler, Applebot, which has fueled speculation that it might be looking to build its own search engine.
But indexing has always been a challenge for companies without deep pockets.
The privacy-minded search engine DuckDuckGo decided to stop crawling the entire web more than a decade ago and now syndicates results from Microsoft. It still crawls sites like Wikipedia to provide results for answer boxes that appear in its results, but maintaining its own index does not usually make financial sense for the company.
“It costs more money than we can afford,” said Gabriel Weinberg, chief executive of DuckDuckGo. In a written statement for the House antitrust subcommittee last year, the company said that “an aspiring search engine start-up today (and in the foreseeable future) cannot avoid the need” to turn to Microsoft or Google for its search results.
When FindX started to develop an alternative to Google in 2015, the Danish company set out to create its own index and offered a build-your-own algorithm to provide individualized results.
FindX quickly ran into problems. Large website operators, such as Yelp and LinkedIn, did not allow the fledgling search engine to crawl their sites. Because of a bug in its code, FindX’s computers that crawled the internet were flagged as a security risk and blocked by a group of the internet’s largest infrastructure providers. What pages they did collect were frequently spam or malicious web pages.
“If you have to do the indexing, that’s the hardest thing to do,” said Brian Schildt Laursen, one of the founders of FindX, which shut down in 2018.
Mr. Schildt Laursen launched a new search engine last year, Givero, which offered users the option to donate a portion of the company’s revenue to charitable causes. When he started Givero, he syndicated search results from Microsoft.
Most large websites are judicious about who can crawl their pages. In general, Google and Microsoft get more access because they have more users, while smaller search engines have to ask for permission.
“You need the traffic to convince the websites to allow you to copy and crawl, but you also need the content to grow your index and pull up your traffic,” said Marc Al-Hames, a co-chief executive of Cliqz, a German search engine that closed this year after seven years of operation. “It’s a chicken-and-egg problem.”
In Europe, a group called the Open Search Foundation has proposed a plan to create a common internet index that can underpin many European search engines. It’s essential to have a diversity of options for search results, said Stefan Voigt, the group’s chairman and founder, because it is not good for only a handful of companies to determine what links people are shown and not shown.
“We just can’t leave this to one or two companies,” Mr. Voigt said.
When Mr. Maril started researching how sites treated Google’s crawler, he downloaded 17 million so-called robots.txt files — essentially rules of the road posted by nearly every website laying out where crawlers can go — and found many examples where Google had greater access than competitors.
ScienceDirect, a site for peer-reviewed papers, permits only Google’s crawler to have access to links containing PDF documents. Only Google’s computers get access to listings on PBS Kids. On Alibaba.com, the U.S. site of the Chinese e-commerce giant Alibaba, only Google’s crawler is given access to pages that list products.
This year, Mr. Maril started an organization, the Knuckleheads’ Club (“because only a knucklehead would take on Google”), and a website to raise awareness about Google’s web-crawling monopoly.
“Google has all this power in society,” Mr. Maril said. “But I think there should be democratic — small d — control of that power.”
New feature can deep-link to specific text on a Web page, with highlighting.
Google has been cooking up an extension to the URL standard called “Text Fragments.” The new link style will allow you to link not just to a page but to specific text on a page, which will get scrolled to and highlighted automatically once the page loads. It’s like an anchor link, but with highlighting and creatable by anyone.
The feature has actually been supported in Chrome since version 80, which hit the stable channel in February. Now a new extension from Google makes it easy to create this new link type, which will work for anyone else using Chrome on desktop OSes and Android. Google has proposed the idea to the W3C and hopes other browsers will adopt it, but even if they don’t, the links are backward-compatible.
The syntax for this URL is pretty strange looking. After the URL, the magic is in the string “#:~:text=” and then whatever text you want to match. So a full link would look like this:
If you copy and paste this into Chrome, the browser will open Wikipedia’s cat page, scroll to the first text that matches “Most breeds of cat have a noted fondness for sitting in high places,” and will highlight it. If the text doesn’t match anything, the page will still load. Backward-compatibility works because browsers currently support the number sign (#) as a URI fragment, which usually gets used for anchor links that are made by the page creator. If you paste this into a browser that doesn’t support it, the page will still load, and everything after the number sign will just be ignored as a bad anchor link. So far, so good.
One problem is that this means you can have spaces in a URL. On a webpage or forum, you can hand-code the link with a href tag (or whatever the non-HTML equivalent is) and everything will work. For instant messengers and social media though, which don’t allow code and use automatic URL parsers, things get a bit more complicated. Every URL parser treats a space as the end of a URL, so you’ll need to use percent-encoding to replace all the spaces with the equivalent “%20.” URL parsers now have a shot at linkifying this correctly, but it looks like a mess:
Spaces aren’t the only characters that can cause problems. The standard RFC 3986 defines several “reserved” characters as having a special meaning in a URL, so they shouldn’t be in a URL. Web-page-authoring tools tend to handle these characters automatically, but now that you’re embedding arbitrary sentences in a URL for highlighting, there’s a higher chance you’ll run into one of these reserved characters:! * ‘ ( ) ; : @ & = + $ , / ? # [ ]. They all need to be percent-encoded in order for the URL to work, and Google’s extension takes care of that for you.
Google’s new Chrome extension, called “Link to Text Fragment,” (it’s also on Github) will put a new entry in Chrome’s right-click menu. You just highlight text on a page, right-click it, and hit “Copy link to selected text.” Like magic, a text fragment link will end up on your clipboard. All the text encoding is done automatically, so the link should work with most websites and messengers.
Google seems like it is going to start pushing out support for text fragments across its Web ecosystem, even without the W3C. The links have already started to show up in some Google search results, which allow Chrome users to zip right to the relevant text. It’s probably only a matter of time before link creation moves from an extension to a normal Chrome feature.
Some of its employees tried to stop their company from doing work they saw as unethical. It blew up in their faces.
00:05very excited to introduce Rana Rana Zay00:08is the global business columnist and00:11natural times and CNN’s global economic00:13analysts previously she’s been the00:15assistant managing editor in charge of00:16Business and Economics at time as well00:18as the magazine’s economic columnist and00:20spent 13 years at Newsweek as an00:22economic foreign affairs editor and00:24correspondent and in her new book don’t00:26be evil which i think is a great title00:29Rhonda Chronicles how far big tech has00:31fallen from its original vision of free00:33information and digital democracy00:35drawing on nearly 30 years of experience00:36reporting on the technologies sector00:39Ronna traces the evolution of companies00:40such as Google Facebook Apple Amazon00:46into behemoths that monetize people’s00:49data spread misinformation and hate00:51speech and threaten citizens privacy she00:53also shows how we can fight back by00:55creating a framework that both fosters00:57innovation and protects us from threats00:59posed by digital technology her book is01:02already garnering widespread praise with01:04the Guardian calling it a masterly01:05critique of the internet pioneers who01:07now dominate our world so without01:08further ado please help me in welcoming01:10Rana for a heart to politics and prose01:16thank you I am so honored to be here01:19it’s really a pleasure this is one of my01:22favorite bookstores probably my favorite01:24bookstore in Washington and so it’s just01:27a huge pleasure I thought I would start01:30by just talking a little bit about how I01:32got the idea to write this book it’s01:33actually my second book my first book01:35makers and takers was a look at the01:38financial sector and how it no longer01:40serves business so I like to kind of01:42take on these big industry-wide maybe01:45take down so we’ve the word but kind of01:49look at an ecosystem and economic01:50ecosystem see how it’s working or not01:52working I got the idea for this book01:56probably two months into my new job at02:00the Financial Times02:01I was hired in 2017 to be the chief02:05business commentary writer so my my job02:08was to sort of look at the top world’s02:11business stories economic stories and02:13try to make sense of them in commentary02:14and when I do that I tend to try and02:17follow the money in order to narrow the02:18funnel of where to put my focus and I02:20had come across a really really02:22interesting statistic that 80%02:25of the world’s wealth corporate wealth02:27was living in 10% of companies and these02:30were the companies that had the most02:31data personal data and intellectual02:34property and so the biggest of those02:36were the big tech platforms that my my02:38book kind of tries to make icons of02:41we’re using all the candy colors here02:43the fangs Facebook Amazon Apple Netflix02:47Google so that was a pretty stunning02:50statistic and it was interesting because02:51I was thinking about how wealth since02:542008 had transferred from the financial02:56sector into the big tech sector and that02:59had happened really quietly without a03:02whole lot of commentary in the press now03:05at the same time I was starting to kind03:06of dig into this story something else03:08happened a much more personal episode I03:11came home one day and I there was a03:14credit card bill waiting for me and I03:16opened it up and I started looking03:17through and there were all these tiny03:19charges in the amount of dollar03:21ninety-nine three dollars five dollars03:22whatever and I noticed that they were03:25all from the app store and I thought oh03:28my gosh I must have been hacked and then03:30I thought who else has my password my03:33ten-year-old son Alex I see nods from03:38parents and others so I go downstairs03:42and I find Alex on the couch with his03:44phone which is his usual after-school03:46position and I say you know what what’s03:50up do you know anything about this and03:51he sort of stunned and oh yes oh that03:55yeah and turns out alex has gotten very03:58fond of a game called FIFA Mobile which04:01is an online soccer game and it’s one of04:03these games that’s dude that you can04:04download it for free but once you get04:07into the game and start playing you have04:10to buy stuff04:11in-app purchases it’s called our loot04:13boxes is another another name so if you04:16want to move up the rankings and do well04:18in the game04:19you have to buy virtual Ronaldo or some04:22new shoes for your player and nine04:24hundred dollars and one month later Alex04:27was at the top of the rankings but I was04:32horrified I was actually horrified and04:34fascinated in fact I mean as04:36mother I was horrified his phone was04:39immediately confiscated passwords were04:41changed limitations were put into place04:44by the way he now officially is allowed04:47only one hour a day on his phone he’s 1304:51years old the average for that age is 704:54hours a day national average now he04:57sneaks in an extra I think he probably04:58gets about 90 minutes because I can’t05:00police him all the time on the way to05:01the on the way to school but it’s I mean05:04to me that is a stunning fact that the05:06average American 13 year old spent 705:09hours day on their phone anyway so I was05:12horrified as a parent but I was05:13fascinated as a business writer because05:15I thought this is the most amazing05:17business model I have ever seen and I05:20have to learn everything about it and05:22right about that time someone had come05:26to see Mia a man named Tristan Harris05:28who’s one of the characters in my book05:29and Tristan is a really interesting guy05:32he was formerly the chief ethics officer05:35at Google and he was trying to bring05:39goodness and not evil to the company and05:42make sure that all the all the products05:45and services were functioning sort of a05:47human interest and then he realized he05:48was not having any luck doing that05:49within the company so he decided to go05:52outside and start something called the05:54Center for Humane technology and Tristan05:57had become really really worried about05:59the core business model that is it’s06:02particularly relevant for Google and06:05Facebook but is also a big part of06:07Amazon’s model and and it’s really the06:08model that another author Shoshanna06:10Zubov who recently wrote a wonderful06:12book on this topic would call06:14surveillance capitalism and so it’s the06:16idea of companies coming in and tracking06:20everything you are doing online and06:22increasingly offline you know if you06:24have your if you have an Android phone06:25it might know where you are in the06:27grocery store if you’re in a car with06:29smart technology your your location06:32coordinates can be tracked so all of06:35this is serving to build a picture of06:37you that is then used to be sold to06:41advertisers and then you can be targeted06:44with what’s called hyper targeted06:46advertising which is essentially why for06:49example06:50if I go online to look for a hotel in06:53California I might get a certain price06:55but someone else might get a different06:57price so this is a really important06:59thing we are looking at different07:01internets right there are subtle07:04differences but they’re there and this07:06data profile that is being built up is07:08splitting us as individual consumers but07:12I would argue that it’s also splitting07:14us as citizens and I’ll when I get to07:16the readings I’ll kind of flush that out07:18a bit more but Tristan07:20kind of turned me on to this business07:23model and he also helped me connect the07:25dots between this business model and07:27what had happened to my son because it07:29turns out that the technologies these07:31sorts of nudges that take you down a07:34game or that bring you to certain places07:36on Amazon or that give you a certain07:39kind of search result or purchasing07:41option on Google are part of an entire07:45field called capped ology which is kind07:49of an Orwellian word and these these07:52technologies actually come largely out07:54of something called the Stanford07:55persuasive technology lab so there is an07:58entire industry that is designed to08:01track your behavior and pull in things08:03like behavioral psychology casino gaming08:06techniques and then layer those on to08:09apps that will push you towards making08:13purchasing decisions or perhaps even08:15other kinds of decisions political08:16decisions that might be good for certain08:19actors and it’s interesting because when08:22I started to think about all this one of08:24the things I really wanted to do in this08:26book was to cry try and create a single08:28narrative arc to take folks through this08:3120 year evolution of this industry from08:34the mid-1990s which is really when the08:36consumer internet was born till now and08:39at the time I was writing and and still08:41probably today you could argue that08:43Facebook was the company that was08:45getting the most negative attention for08:48a lot of the economic and political08:49ramifications of its business model but08:51if you go back to the very beginning08:53Google is the most interesting way to08:56track this because Google really08:59invented the targeted advertising09:01business model they really invented09:03surveillance capitalism and one of the09:05things that is fascinating and and09:06sometimes I’m asked what’s the most09:08surprising thing that you found when09:10writing this book and really the most09:11surprising thing is it was all hiding in09:14plain sight so if you go back to the09:17original paper the Larry Page and Sergey09:19Brin who were the founders of Google did09:21in 1998 while at Stanford as graduate09:25students they actually lay out they lay09:28out what a giant search engine would09:30look like how it would function but then09:31how you might pay for it and if you go09:34down to page 33 there is a section in09:36the appendix called advertising and its09:38discontents and it essentially says that09:42if you monetize a search engine in thisthis way with hyper targeted advertisingthe interests of the users and theinterests of the advertisers be theycompanies or who knows what publicentities are eventually going to comeinto conflict and so they actuallyrecommend that there be some kind ofacademic search engine an open searchengine in the public interest so this to10:05me first of all is fascinating that it10:07was just there all along and fascinating10:11that very few people have read that10:13entire paper even though even those that10:16write about it which in some ways kind10:18of goes to the point that in the last 2010:20years we all do a lot less reading not10:22folks here but but in general we do less10:25reading there was actually a fascinating10:26study that came out recently from common10:28sense media which is Jim’s dyers group10:30in California that tracks children’s10:33behaviors online teenagers only10:36one-third of them read for pleasure more10:39than once a month10:41long-form articles doesn’t matter if10:43you’re reading on an e-book or device10:44but long-form articles books only once a10:47month for pleasure so all our entire10:50world has been changed economically10:52these companies have huge monopoly power10:54politically we’re all kind of living10:56with the ramifications of this new world10:58of social media disinformation fake news11:01and cognitively our brains are changing11:05our behaviors are changing so connecting11:07all of those things was really what I11:10was trying to get at in this book and so11:13I’m gonna read two or three maybe short11:16excerpt11:17and then we can leave a lot of time for11:19questions so that people can kind of11:20dive into as much of this as they want11:23and I’ll start perhaps with my very11:28first meeting with the Googlers Larry11:33Page and Sergey Brin who I met not in11:36Silicon Valley but in Davos the Swiss11:39gathering spot of the global power elite11:42where they had taken over a small Chalet11:44to meet with a select group of media the11:47year was 2007 the company had just11:50purchased YouTube a few months back and11:52it seemed eager to convince skeptical11:54journalists that this acquisition wasn’t11:56yet another death blow to copyright paid11:58content creation and the viability of12:00the news publications for which we12:02worked12:02unlike the buttoned-up consulting types12:05or the suited executives from the old12:07guard multinational corporations that12:09roamed the promenades of davos their12:11tasseled loafers slipping on the icy12:13paths the Googlers with a cool bunch12:15they wore fashionable sneakers and their12:17chalet was sleek white and stark with12:19giant cubes masquerading as chairs in a12:21space that looked as though it had been12:23repurposed that morning by designers12:25flown in from the valley in fact it may12:27have been and if so Google would not12:29have been alone in such access I12:30remember attending a party once in Davos12:32hosted by Napster founder and former12:34Facebook president Sean Parker that12:37featured giant taxidermy bears and a12:39musical performance by John Legend back12:42in the Google Chalet Brin and page12:44projected a youthful earnestness as they12:46explained the company’s involvement in12:48or authoritarian China and insisted12:50they’d never be like Microsoft which was12:52considered the corporate bully and12:53monopolist at the time what about the12:55future of news we wanted to know after12:57admitting that page read only free news12:59online whereas Brin often bought the13:01sunday New York Times in print it’s nice13:03he said cheerfully13:04the duo affirmed exactly what we13:07journalists wanted to hear Google they13:09assured us would never threaten our13:10livelihoods13:11yes advertisers were indeed migrating13:14and mass from our publications to the13:15web where they could target consumers13:17with a level of precision that the print13:19world could barely imagine but not to13:21worry Google would generously retool our13:22business models so we too could thrive13:24in the new digital world I was much13:27younger than and not the admittedly13:29cynical business journalist that I have13:30since13:31and yet I listened skeptically13:32skeptically to that happy future of news13:35like lecture whether Google actually13:37intended to develop some brilliant new13:40revenue model or not what alarmed me was13:42that none of us were asking a far more13:44important question sitting towards the13:46back of the room somewhat conscious of13:48my relatively junior status I hesitated13:50waiting until the final moments of the13:52meeting before raising my hand excuse me13:55I said we’re talking about all this like13:57journalism is the only thing that13:58matters but isn’t this really about13:59democracy if newspapers and magazines14:02are all driven out of business by Google14:04or companies like it I asked how are14:06people gonna find out what’s going on14:07Larry Page looked at me with an odd14:10expression as if he were surprised that14:11someone should be asking such a naive14:13question oh yes we’ve got a lot of14:16people thinking about that14:17not to worry his tone seemed to say14:19Google had the engineers working on that14:22little democracy problem next question I14:26read that because I’m kind of amazed14:30there is still a real lack of14:34understanding I think in the valley14:36about some of the real negative14:39externalities of what have been let’s14:41face it amazing technologies I mean we14:43you know where would we be without14:44search in our smartphones we all14:46carrying around the power of a mainframe14:47in our pockets but as a journalist I14:51think there’s really been a an inability14:54of these companies to kind of own up to14:56you know some of the bad stuff that they14:59have wrought and I think that that still15:00considers oh sorry still continues to be15:03to be the case one of the other points15:06that I try and make in the book is that15:09the problems I’m talking about have15:12actually moved outside of just the big15:14four flat platform firms that that we’re15:16moving into a world in which15:17surveillance capitalism is going to be15:19part of the healthcare system and the15:21financial system and really every kind15:24of business is now using this as its15:26model so for example if you buy coffee15:29at Starbucks Starbucks knows a lot about15:30you Johnson & Johnson knows a lot about15:33you there there are firms watching you15:35all the time and so we’re really at a15:37pivot point I think where we have to ask15:40as a society what are the deeper15:43implications of this and our15:44okay with them and so I would like to15:47read another excerpt where I look at how15:50this model is is moving into the15:52insurance sector and what that means so15:58far data has been obtained via computers16:01and mobile devices but now with the rise16:03of personal digital assistants like16:05Amazon’s Alexa Google’s home mini and16:07Apple Siri now at 30 and now in a third16:10of American homes with triple digit16:12sales growth a year the human voice is16:14the new gold while reports of Alexa16:16Alexa and Siri listening in on16:18conversations and phone calls are16:19disputed there’s no question that they16:21can hear every word you say and from16:23there it’s a short step to them using16:24that knowledge to direct your purchasing16:26decisions it isn’t much of a longer step16:28to see the political implications16:30already some researchers worry that16:32digital assistants will become even more16:33powerful tools than social media for16:36election manipulation certainly none of16:38us will be unaffected consider consider16:41that homeowner oops sorry16:43I’m reading from a reading from the16:44wrong part I think apologies somehow16:54picked the wrong section here anyway I’m16:57going to talk you through this example16:58because it’s it’s something that is17:01already out there I had a conversation a17:03couple of years ago with an executive17:04from Zurich Financial which is a big17:07financial company they do insurance many17:10parts of the world they will now if17:12you’d like them to put sensors in your17:14home or in your car and if you have for17:18example as I do you live in a 190117:20townhouse let’s say you’re upgrading17:22your pipes you get a check you get a you17:24know a positive mark and you may see17:26your insurance premium go down but let’s17:30say your kid is smoking a joint in their17:32bedroom and the sensor picks up on that17:34you then get a black mark here and your17:36premium may go up same again in your car17:39if you’re speeding your insurance17:42company will know and so on and so forth17:43now you can either like this or not17:46depending on where you sit in the17:48socio-economic spectrum but what’s very17:50very interesting is that entire business17:53model a pooled risk business model17:55that’s what insurance is it’s now been17:57completely dissed17:58so you can be targeted and split so this18:02is no longer about society pulling risk18:04a saree pooling risk this is about18:06individuals having to own the risk so if18:09you take that to its natural conclusion18:12you can imagine an elite up here that18:17has access to special pricing and all18:19kinds of great products but you can also18:21imagine an uninsurable group of people18:25at the bottom and then who is going to18:28pick up that risk now the public sector18:30may be maybe they’ll be a junk bond18:33market for insurance either way you have18:36a split in society that didn’t exist18:39before and that was always the business18:42model here you know you go back and read18:44some of the early work of someone like18:46Hal Varian for example who was the chief18:48economist at Google splitting pricing18:51down to the individual was always the18:53point of platform technology firms like18:56Google or Facebook or Amazon splitting18:58individuals out so they could be18:59targeted in different ways but that not19:01only splits pricing it splits Society19:05and so that’s kind of really the the19:07core issue I want to get out here19:10I think I’ll maybe read just just one19:13more excerpt and then we can do we have19:15we have time yeah and then we’ll open it19:17up for questions after that my first19:22book just to mention again was about the19:25financial industry and one of the things19:26that strikes me is that big tech19:28companies have in some way become the19:30new too big to fail entities not only19:33are they holding more wealth and power19:35than the largest banks but in some ways19:36they function like banks they have a19:39tremendous amount of money they use it19:41to buy up corporate debt if that debt19:44were to go bad that could actually be19:46the beginnings of another financial19:47crisis and so that’s kind of a part of19:49this story that really hasn’t gotten out19:51there so let me let me read just two or19:54three more pages for you on that topic19:57the late great management guru Peter20:00Drucker once said in every major20:01economic downturn in US history the20:03villains have been the heroes during the20:05preceding boom I can’t help but wonder20:08if that might be the case over the next20:10few years as the you know20:11it states and possibly the world heads20:13towards its next big slowdown downturns20:16historically come about once every20:18decade and it’s been more than that20:19since the 2008 financial crisis back20:22then banks were the too-big-to-fail20:24institutions responsible for our falling20:26stock portfolios home prices and20:28salaries technology companies by20:30contrast have led the market upswing20:32over the past decade but this time20:34around it’s the big tech firms that20:36could play the spoiler role you wouldn’t20:39think that it could be so when you look20:40at the biggest and richest tech firms20:42today take Apple for example warren20:44buffett says he wished he owned even20:45more Apple stock Goldman Sachs is20:47launching a new credit card with the20:48tech Titan which became the world’s20:50first trillion-dollar market cap company20:52in 2018 but hidden within these bullish20:55headlines are a number of disturbing20:57economic trends of which Apple is20:59already exemplar study this one company21:02and you begin to understand how big tech21:04companies the new too-big-to-fail21:05institutions could indeed sow the seeds21:08of the next financial crisis the first21:11thing to consider is the financial21:12engineering done by such firms like most21:15of the largest and most profitable21:17multinational companies Apple has loads21:19of cash about 300 billion as well as21:22plenty of debt close to 122 billion21:24that’s because like nearly every other21:27large rich company it has parked most of21:30its spare cash in offshore bond21:32portfolios over the last ten years at21:34the same time since the 2008 crisis is21:37that it is issued cheap debt at rates to21:41do sorry it is issued cheap rate sorry21:44cheap debt at low rates in order to do21:48record amounts of share buybacks and21:50dividends Apple’s responsible about a21:53quarter of the 407 billion in buybacks21:55and out since the Trump tax bill was21:57passed in December of 2017 but buybacks22:00have bolstered mainly the top 10% of the22:03US population that owns 84% of all stock22:06the fact that share buybacks have become22:08the biggest single use of corporate cash22:10for over a decade now has buoyed markets22:13but it’s also increased the wealth22:15divide which many common economists22:17believe is that not only the single22:19biggest factor in slower than historic22:21trend growth but is also driving22:22political populism which threatens the22:25good system itself that phenomenon has22:28been put on steroids by the rise of yet22:30another trend epitomized by Apple22:33intangibles such as intellectual22:35property and brands now make up a much22:37larger share of wealth in the global22:39economy the digital economy has a22:41tendency to create super stars since22:43software and internet services are so22:45scalable and they enjoy network effects22:50let’s see do but as these as software22:56and internet services become a bigger22:58part of the economy they reduce23:00investment across the economy as a whole23:02and that’s not only because banks are23:03reluctant to lend to businesses whose23:06intangible assets may simply disappear23:08if they go belly-up but because of the23:10winner-take-all effect that a handful of23:12companies including Apple Amazon and23:14Google enjoy so to sum this up in plain23:17English as this handful of companies has23:20gotten bigger and more powerful23:21investment in the overall decline23:23economy has declined the number of jobs23:26that they’re creating relative to their23:28market size is much lower than that in23:30the past so you have the superstar23:32economy that has become kind of a23:33winner-take-all game I think that we’re23:37going to probably see some kind of a23:39market correction in the next couple of23:41years it’s going to be very interesting23:43at that point to see whether tech leads23:45the markets down and whether you might23:47then see a kind of an Occupy Silicon23:49Valley sentiment as you did in 2008 with23:53Occupy Wall Street I think that that’s23:54really quite possible we can delve more23:57into that if you’d like but I think I23:59want to stop here and be respectful of24:01question time and there are parts that24:04you guys want to hear more about or24:06particular areas that I could read more24:08from you can let me know go ahead24:15because sure we don’t get to speak very24:18often you and I one is you’ve doubtless24:23read about Bloomberg’s decision recently24:26to forbade its reporters from covering24:28Michael Bloomberg yeah yet The24:31Washington Post has no problem24:34investigating Vsauce do you see is that24:38a problem for you have you thought about24:40that is that a and so have any24:43consistency that should bother at24:45financial journalists and the second24:46question is how important for any24:51solution to the problems you you raise24:53would an tights for the revival of24:56antitrust be s we see on the continent24:59where it’s more aggressive and among25:01some of the the Democratic candidates25:04for the president well so let me take25:06the antitrust question first that’s25:08actually important part of the book25:10there’s an entire chapter on antitrust25:12and I think we probably are gonna see25:15some shifts as folks may know since the25:191980s onward antitrust in America has25:23basically been predicated on price so as25:25long as consumer prices were falling it25:28was perceived that companies could be as25:30big as they wanted that it wasn’t a25:31problem but one of the things I look at25:34in the book is this this shift to a25:36world in which transactions are being25:39done not in dollars but in data so25:42that’s a that’s a barter transaction25:43really and one of the things that’s so25:45interesting and this is actually a way25:47in another way in which Silicon Valley25:49is similar to Wall Street the25:50transaction is really opaque so you25:53don’t know essentially how much you’re25:55paying for the supposedly free service25:58that you’re receiving that is a very26:02difficult market to create fairness26:04within and it probably makes the Chicago26:07School notion of consumer prices going26:10down no problem I think probably26:13irrelevant and so there’s two ways in26:15which that’s being dealt with you have26:17the rise of this new Brandeis school of26:19thinking in which you know maybe this is26:22really about power maybe maybe we should26:25think about the big tech firms26:26we do the nineteenth-century railroads26:28we’re alright you know you had at one26:30point railroad Titans that would come in26:33and build tracks and then own the cars26:35and then own the things that were in the26:37cars and eventually that became a26:39zero-sum game and it’s you know it’s as26:42folks probably know we’re in a period in26:44which there’s as much concentration of26:46wealth and power as there was in the26:47Gilded Age so I could imagine very26:50easily a scenario in which you could26:51justify Amazon say being the platform26:55for e-commerce but not being able to26:57compete in the specific areas of fashion27:01or you know whatever else they’re27:03selling against other customers and in27:05fact that’s already the case in the27:06financial sector that big companies that27:09trade let’s say aluminum you know as27:12Goldman Sachs did this is what it ran27:14into a suit a few few years ago that it27:16was both owning all the aluminum and27:18trading it and that’s that’s27:20anti-competitive and so that became an27:22issue for the Fed so I think we probably27:24are going to see that kind of ruling as27:26for the post and journalism you know27:29it’s funny I have some friends that are27:30they’re quite influential to post and27:34they say that Bezos is pretty hands-off27:37I mean I can’t I can’t vouch for that27:38one thing I will say is that Amazon did27:41put this book on the top 20 nonfiction27:43what Stern’s a month so you know I don’t27:46know if that’s a ploy to make me think27:48that they’re they’re being really fair27:49but from probably Jeff Bezos I don’t27:52know I he probably not thinking that27:53much about this book or me but anyway27:56next question go ahead so it seems like27:59some of the major decisions that these28:01big tech companies are making are in28:04regard to fake news and how they’re28:06moderating fake news or the lack of it28:08so have you seen maybe an approach by28:11any current social media platform or any28:13proposed plans in place that you think28:15would be best for moderating fake news28:17that’s such a good question so just to28:20kind of pull back the the two points of28:22view on that are hey look you know the28:26platform tech companies are essentially28:27giant media and advertising firms right28:30I mean if you look at the business model28:31of a Google or a Facebook it’s28:34essentially just like the Financial28:35Times or CNN it’s just much more28:37effective and it can be targeted to the28:39individual28:40that means that these firms have taken28:42you know 85 90 percent of the app new28:45digital advertising pie in the last few28:47years now given that they function as28:49media companies should they not be28:51liable for disinformation in the way28:55that a media company would be so if I28:57print something incorrect at the FT28:59that’s you know the the paper and also29:02my hide on the line there I think that29:05we should actually think about rolling29:08back some of those loopholes that these29:09firms enjoy since the mid-1990s onwards29:12I think that they are going to have to29:14take some responsibility now the29:16question is do we want Mark Zuckerberg29:18being the minister of truth and that’s29:20that’s that’s a really tough question29:23what I would prefer is for the29:26government to actually you know for29:28democratically elected governments to29:29come up with rules about what is and29:32isn’t appropriate and to not have29:34individual companies making those29:36choices I think we’re in a period right29:38now where you know you’ve got Twitter29:40you’ve got Google to a certain extent29:41coming out saying okay we recognize we29:43need to do things differently that’s29:44putting pressure on Facebook but at the29:46end of the day we’re gonna have to have29:47I think an entirely new framework not29:51just in this area but also in taxation29:53in you know an antitrust which we’ve29:56already talked about this is the shift29:58that we’re going through is I think the30:00new Industrial Revolution it’s a 70 year30:03transition and it’s going to require a30:04lot of different frameworks relative to30:07what we already have so the answer is no30:10I don’t see any particular company that30:12has come up with the right framework yet30:14any other questions30:16oh yeah I’d like to go back to antitrust30:18for a minute the Washington Post put up30:20an article just this afternoon about how30:23Apple is changing its business model and30:25it’s different as you know it’s30:27differentiated itself in the market by30:29saying they care about privacy well now30:32they are moving from a a device company30:38to a services company according to the30:40article and they are used and they are30:43using privacy as a lever to provide30:46services that their that other smaller30:51companies like tile which is the example30:54the article has used to create a market30:59for itself right and so it says in the31:03article that the feds are considering31:04looking at antitrust measures against31:06Apple but I think it raises a bigger31:09question that you pointed to which is31:13that the models of antitrust don’t work31:16anymore so in terms of privacy lots of31:22people have talked about monetizing31:24privacy getting paid yeah data but how31:27do you think from an economic point of31:30view we as a society need to look at the31:33role of privacy and the role of31:35antitrust together to somehow change the31:38way we think about these companies31:41because in addition we’ve got31:43consolidation in the marketplace so yeah31:45no longer fair competition you can’t31:48become another Amazon right easily31:51because there are so many big so mate31:53because the players are big and there31:55are so few of them in each part of the31:58economy yeah a right so there’s a lot in32:00what you’ve just said for starters I32:03think you’re hitting on something really32:04important which I get at in my solutions32:06chapter that this is such a huge shift32:09and it’s touching so many different32:11areas and we’ve talked about privacy32:13we’ve talked about antitrust we haven’t32:15even gotten into national security you32:17know civil liberties I mean there there32:19are so many different areas and when you32:21one of the things I noticed when I sat32:23down to write the solution sections you32:25know when you do a think book you always32:26have to have the solutions section and32:28you know the publisher wants like that32:29Silver Bullet thing and you look at this32:32and you notice that when you pull a32:33lever here it effects something in this32:35other areas so I think that’s one reason32:38why we should have a national committee32:41to actually look at what are all the32:43questions it’s when I speak to folks32:45particularly in DC policymakers there’s32:47you know the antitrust camp here the32:49privacy camp here the security folks32:50there that conversation needs to be32:52happening in a 360 way and it is32:54happening much more so that way in32:57Europe I will say I just came off of two32:59weeks of book touring in Europe and the33:02conversation there I think is much more33:04developed and they seem to be to go to33:06your point about the ecosystem and how33:08share it one of the things that seems to33:11be folks seem to be headed towards is a33:13public digital Commons a kind of a33:16database let’s say alright if you decide33:19as you know the cat seems to be out of33:21the bag that we’re gonna allow33:22surveillance capitalism I mean there33:24there are certain folks like Shoshanna33:26would love to see the dial turned back33:28I’m not sure if that’s possible let’s33:30have a public database in which not just33:33one corporation or a handful of33:35corporations but multiple sized players33:37as well as the public sector as well as33:40individual citizens who’s you know after33:43all it’s our data being harvested33:45everybody gets access and then you can33:47figure out how you want to share the pie33:49and one interesting example recently is33:51the Google sidewalk project in Toronto33:54it sounds like you’re up on these issues33:56so you’re probably aware but Google had33:59taken over sort of twelve acres on the34:01Toronto Waterfront and put sensors34:04everywhere and the idea was to create a34:06smart city in which you’d be able to34:08manage traffic patterns and energy usage34:10and things like that but until recently34:12Google was going to own all that data34:14and have access to and finally the34:16Toronto government got a clue and said34:18well actually you know what let’s put34:19this in a public database so other34:22smaller or midsize local firms can come34:25in and be part of that economic34:26ecosystem but also as a public sector we34:30can decide well maybe we want to share34:32data for energy issues or for health34:36issues but maybe we don’t want to share34:37it for certain other kinds of things and34:40perhaps there would be some way in which34:42individuals could take back some of that34:44value so California is thinking about a34:47digital dividend payment from the big34:49tech companies there’s also been talk of34:51a digital sovereign wealth fund if you34:53think about kind of data as the new oil34:56whatever the value is judged to be it34:59would be putting the sovereign wealth35:01fund in the same way that Alaska or35:02Wyoming give back payments or use that35:05for the the public sector that could be35:08done with data too so I think something35:10like that is probably going to be the35:12best solution I’ll tell you I have many35:14examples in the book of ways in which35:16the bigger players have been able to35:18squash small and mid-sized firms and35:20that35:21a major issue and a lot of venture35:23capitalists that I speak to are actually35:26becoming concerned about that because35:27they say that there’s sort of black35:29zones of innovation where if Amazon is35:33there or Google is there you really35:34can’t start a business there’s just been35:36too much that’s been been written35:38ring-fenced question over here35:40yes while your book may be the the best35:43one on the subject they’ve certainly35:44been other books before talking about35:46individuals privacy and their their data35:49and everything about them why is it that35:52you think people are so unconcerned35:56about handing over all of their data to35:58these companies when they are perhaps36:00very concerned about handing it over to36:02the government why why do they feel36:04these guys are the good guys and the36:06government is necessarily the bad guys36:08yeah it’s such an interesting question36:11and that really varies from country to36:13country I find that that’s sort of an36:15interesting cultural dynamic that can36:17shift depending on what market you’re in36:19I have really been puzzled as to why36:23people are so first of all why everybody36:25just clicks the box and says no problem36:27I think part of that is is the opacity36:29of the market I mean if you kind of go36:31back to Adam Smith basic economics you36:35need three things to make a market36:36function property properly that would be36:38equal access to data transparency in the36:41transaction and a shared moral framework36:44and you could argue that none of those36:46things are in place so when we’re making36:48these transactions I think as that’s36:51that very fact becomes better explained36:56and people begin to kind of understand36:58that narrative like the insurance36:59example I just gave that all right37:02you’re getting something but you’re37:03giving up a lot I’m beginning to see37:07pushback already and I suspect in recent37:10weeks as some of the big players have37:11moved into healthcare you know into into37:14the commercial banking business I just37:17think that we are going to begin to see37:18more people being reluctant to give up37:23that much value for what they’re getting37:25you’re also interestingly seeing when37:28there are other options people will go37:31elsewhere so Jimmy Wales who started37:33Wikipedia just I think37:34the weeks ago came up with a new social37:36networking site he’s already got 300,00037:38users there and it’s an odd37:41they don’t do targeted advertising it’s37:42run on the wiki model where you can37:44donate if you want I think once the37:47antitrust piece is in place and you37:49actually have space for new competitors37:52to come in and to offer up different37:54kinds of services that perhaps are more37:56respectful of privacy that you you know37:58you could see a shift there but I’m38:00curious actually can I pull the audience38:02for a minute because I want to ask how38:04many people think that in the next five38:07years individuals are going to become38:09more worried about giving up information38:12that’s going to change their behavior38:13online so like two-thirds but not yeah38:19that’s interesting okay oh go ahead38:23sorry we’re sheep we’re cheap oh my god38:26that was a different book curious if you38:30see the administration’s38:32suggestion that it the California can’t38:35set its own rules for gas mileage and so38:41on and emissions as having a parallel in38:44this area you know I hadn’t thought38:48about that question before I always38:50think about California as really being38:53very leading what is eventually going to38:55become the national standard and I think38:58in data I feel like that is gonna happen39:01you know even the Europeans in fact are39:04saying that the California model is39:06probably the better model for data data39:08protection and privacy and sharing of39:11value so the Europeans have GDP are you39:13know which was kind of the first step in39:15the privacy direction but it doesn’t39:17take into account that economic39:18ecosystem so perversely you have the big39:21companies maybe being able to do better39:23with the GDP our model and smaller ones39:26getting cut out of the loop because they39:27don’t have the legal muscle to kind of39:29deal with all the rules so I do think39:31the California model is going to become39:32a de facto standard we also haven’t39:34talked about China which is of course39:36going its own way and I have it I have a39:38chapter in the book where I look at that39:40I look at the current trade war tech war39:43kind of through the lens of surveillance39:44capitalism and39:46that’s gonna be very interesting I think39:48one of the big probably the biggest mid39:52to long-term economic question for me is39:54are we going to see a transatlantic39:56alliance around digital trade and coming39:59up with some standards because China is40:01going its own direction it’s going to40:02develop its own ecosystem it has its own40:04big players the u.s. is in another40:07category but where is Europe gonna be is40:09it going to be a tri polar world is it40:11going to be a bipolar world in terms of40:13how all this works that that’s a major40:15ik you cannot make an actually foreign40:16policy question I think hey thanks for40:21coming and thinking um I’m wondering we40:25have like a Department of Agriculture we40:27have a Department of Energy will there40:29be a Department of Technology ever in40:30the US and which other countries already40:33have that kind of thing going yeah40:35England is talking about that actually I40:37think kind of an FDA of Technology is40:40probably a very good idea you know I see40:44going back to the example about my son40:46there there40:46the research is nascent and causality is40:49is difficult to prove but there there’s40:51you know a new body of research since40:542011 2012 when smartphones really became40:57ubiquitous showing that levels of40:59anxiety and depression and younger41:01people arising you know they’re there41:04they’re issues of self harm sometimes41:07when people you know use these41:08technologies addictively so I think that41:11that’s that’s a big issue to me it’s41:12very similar to cigarettes you know41:14those were regulated there was a41:16different narrative and then behaviors41:17changed and I think I think that that’s41:20one area to consider policy wise there41:25may be time for one or two more41:26questions41:27okay sorry over here and then over here41:29hi41:30I’m kind of curious what you think about41:33the fact that most of these41:36conversations around technology or even41:38democracy tends to focus on institutions41:41and systems and structures which is41:44great because they are so powerful and41:47ubiquitous my background is in teaching41:51critical thinking and in conflict41:54management and I what I worry41:58that so little attention is being paid42:01to the intelligence and maturity of the42:05citizenry I’m from India after 70 years42:10of democracy we’ve lost it I think it’s42:15simplistic to blame the right-wing42:18leaders and the government I believe we42:22as a people have not developed the42:24maturity to be effective intelligent42:30citizens we don’t have the values we are42:34still feudal we are still extremely42:37hierarchical we don’t have the42:40democratic values in India and we didn’t42:43cultivated over 70 years I see a42:46parallel to being susceptible to the42:52seductions of Technology whether it be42:56free news or the click baiting or43:00anything that the big companies seduce43:04us with that even as we need as you said43:08they an FDA kind of for technology we43:13seem to be observing ourselves of the43:17responsibility of being you know of43:20waking up and no se pians I hear I hear43:24what you’re saying and it’s interesting43:25two things come to mind first of all as43:28I say I just got back from Europe the43:29debate is much more nuanced there and43:33and further along and I think that’s in43:36part because there was not quite as much43:39pendulum shift in the last 40 or 5043:41years from the public sector to the43:43private sector as there was here I think43:46I’m not quite sure if I agree entirely43:49with your point about institutions I43:50think in some ways part of the problem43:53one of the reasons why we have43:54concentration levels that are same as43:57they were in the 19th century is that44:00you know we have a generation of44:03business leaders that grew up in the 80s44:04thinking that the government was only44:06good for cutting taxes and there’s hyper44:09individualism that’s that’s44:12the entire economic model and in some44:14ways I think that you know Facebook is44:16maybe the apex of the neoliberal44:19economic model if you think about the44:22problems of globalization were that cap44:25but you know it was supposed to be44:27globalization was supposed to be about44:28capital goods and people crossing44:30borders well it turned out the capital44:31could cross a lot faster than either44:33goods or people if you take that into44:36the world of data that’s even more true44:38and so I think that you have a group of44:41companies now that have really44:44turbocharged a lot of the problems that44:46have given us the politics that we have44:49now and and a company like Facebook I44:51mean I think it every time Zuckerberg is44:52on the hill it’s like there’s this44:54attitude that they are supranational you44:56know and kind of flying 35,000 feet44:59above national concerns and I think that45:02that’s part of a larger shift and45:04probably going to be a big part of the45:052020 debate right are we gonna now have45:08a pendulum shift back away from private45:12power to some public power some45:14different sharing of that which is a45:16values question which I think gets at45:18some of what you’re talking about45:20long-winded answer anyway I think we45:22have time for maybe one more question45:23yeah – quick question okay one is some45:27of the tech companies especially the45:29platform companies have you know why45:32should we not consider looking at them45:35as utility companies yeah I mean we’ve45:39had phone companies and as far as I know45:41they don’t data mine our conversations45:43and maybe mistaken a bit right I mean45:46right that’s they could easily right45:49right yes it’s different different45:50business model yeah yeah so so that was45:52one the other thing is you mentioned45:54that eventually we need tech policy45:56around this and the issue at least my45:59issue is that the people who make these46:01decisions the the policy makers they46:05just most to them don’t have the46:07technical background right to properly46:10assess the different choices and make46:12those decisions I mean I think one of46:15them Zuckerberg or someone testified the46:17questioning was just awful I mean they46:20just ignore our tech support was46:22terrible46:23yeah exactly so I know anyway whatever46:27thoughts you have no that’s a great and46:29that’s like maybe a great place to sort46:31of wrap up I think the utility model is46:34completely viable and it’s interesting46:36one of the bits of pushback that you’ll46:38sometimes get from folks in the valley46:40about that is well if we’re if we’re46:42split in this way or if the the capacity46:46to innovate is sort of you know46:47compressed as the profit margins would46:49be compressed in a utility model that’ll46:52be bad for innovation not really I mean46:54there’s the statistics show for starters46:56that companies innovate more when46:58they’re smaller they tend to innovate47:00more when they’re private and breakups47:03in the past you can argue have actually47:05created more innovation so a lot of47:07academics would say that even the the47:10the the antitrust just the threat of47:13antitrust action against Microsoft was47:15one of the reasons that Google was47:16allowed to to blossom as it did you can47:19go back to the breakup of the bells and47:22say maybe that created space for the47:25cellphone industry to to move ahead so I47:28think there’s a lot of examples that a47:31more decentralized model is actually a47:34good thing and I think that that is47:36actually going to be a really important47:37thing because right now there’s this I47:40think very perverse debate in the u.s.47:42that is bringing together parts of the47:45far right and parts the far left that47:47all right we need these companies to47:48stay big because they’re the national47:50champions and the the becoming war with47:52China that is a complete bunk that is47:56not shown out first of all I mean these47:58companies would love to be in China if47:59they could get into China you know I48:02think decentralized is the advantage in48:06all respects in the US economically so48:09yeah I’m have no problems with a utility48:12model anyway I think my time is up but48:15I’d be happy to sign books and answer48:17any other questions here at the table48:18and thanks so much for your attention48:19[Applause]48:34you
It was the most ambitious library project of our time—a plan to scan all of the world’s books and make them available to the public online. “We think that we can do it all inside of ten years,” Marissa Mayer, who was then a vice-president at Google, said to this magazine in 2007, when Google Books was in its beta stage. “It’s mind-boggling to me, how close it is.”
Today, the project sits in a kind of limbo. On one hand, Google has scanned an impressive thirty million volumes, putting it in a league with the world’s larger libraries (the library of Congress has around thirty-seven million books). That is a serious accomplishment. But while the corpus is impressive, most of it remains inaccessible. Searches of out-of-print books often yield mere snippets of the text—there is no way to gain access to the whole book. The thrilling thing about Google Books, it seemed to me, was not just the opportunity to read a line here or there; it was the possibility of exploring the full text of millions of out-of-print books and periodicals that had no real commercial value but nonetheless represented a treasure trove for the public. In other words, it would be the world’s first online library worthy of that name. And yet the attainment of that goal has been stymied, despite Google having at its disposal an unusual combination of technological means, the agreement of many authors and publishers, and enough money to compensate just about everyone who needs it.
The problems began with a classic culture clash when, in 2002, Google began just scanning books, either hoping that the idealism of the project would win everyone over or following the mantra that it is always easier to get forgiveness than permission. That approach didn’t go over well with authors and publishers, who sued for copyright infringement. Two years of insults, ill will, and litigation ensued. Nonetheless, by 2008, representatives of authors, publishers, and Google did manage to reach a settlement to make the full library available to the public, for pay, and to institutions. In the settlement agreement, they also put terminals in libraries, but didn’t ever get around to doing that. But that agreement then came under further attacks from a whole new set of critics, including the author Ursula Le Guin, who called it a “deal with the devil.” Others argued that the settlement could create a monopoly in online, out-of-print books.
Four years ago, a federal judge sided with the critics and threw out the 2008 settlement, adding that aspects of the copyright issue would be more appropriately decided by the legislature. “Sounds like a job for Congress,” James Grimmelmann, a law professor at the University of Maryland and one of the settlement’s more vocal antagonists, said at the time. But, of course, leaving things to Congress has become a synonym for doing nothing, and, predictably, a full seven years after the court decision was first announced, we’re still waiting.
There are plenty of ways to attribute blame in this situation. If Google was, in truth, motivated by the highest ideals of service to the public, then it should have declared the project a non-profit from the beginning, thereby extinguishing any fears that the company wanted to somehow make a profit from other people’s work. Unfortunately, Google made the mistake it often makes, which is to assume that people will trust it just because it’s Google. For their part, authors and publishers, even if they did eventually settle, were difficult and conspiracy-minded, particularly when it came to weighing abstract and mainly worthless rights against the public’s interest in gaining access to obscure works. Finally, the outside critics and the courts were entirely too sanguine about killing, as opposed to improving, a settlement that took so many years to put together, effectively setting the project back a decade if not longer.
In the past few years, the Authors Guild has usefully proposed a solution known as an “extended collective licensing” system. Using a complex mechanism, it would allow the owners of scanned, out-of-print libraries, such as Google or actual non-profits like the Hathitrust library, to make a limited set of them available with payouts to authors. The United States Copyright Office supports this plan. I have a simpler suggestion, nicknamed the Big Bang license. Congress should allow anyone with a scanned library to pay some price—say, a hundred and twenty-five million dollars—to gain a license, subject to any opt-outs, allowing them to make those scanned prints available to institutional or individual subscribers. That money would be divided equally among all the rights holders who came forward to claim it in a three-year window—split fifty-fifty between authors and publishers. It is, admittedly, a crude, one-time solution to the problem, but it would do the job, and it might just mean that the world would gain access to the first real online library within this lifetime.