openpolitics.com

As computer programmers, our formative intellectual experience is working with deterministic systems that have been designed by other human beings. These can be very complex, but the complexity is not the kind we find in the natural world. It is ultimately always tractable. Find the right abstractions, and the puzzle box opens before you.

The feeling of competence, control and delight in discovering a clever twist that solves a difficult problem is what makes being a computer programmer sometimes enjoyable.

But as anyone who’s worked with tech people knows, this intellectual background can also lead to arrogance. People who excel at software design become convinced that they have a unique ability to understand any kind of system at all, from first principles, without prior training, thanks to their superior powers of analysis.

.. Approaching the world as a software problem is a category error that has led us into some terrible habits of mind.

BAD MENTAL HABITS

First, programmers are trained to seek maximal and global solutions. Why solve a specific problem in one place when you can fix the general problem for everybody, and for all time? We don’t think of this as hubris, but as a laudable economy of effort. And the startup funding culture of big risk, big reward encourages this grandiose mode of thinking. There is powerful social pressure to avoid incremental change, particularly any change that would require working with people outside tech and treating them as intellectual equals.

.. Instead of relying on algorithms, which we can be accused of manipulating for our benefit, we have turned to machine learning, an ingenious way of disclaiming responsibility for anything. Machine learning is like money laundering for bias. It’s a clean, mathematical apparatus that gives the status quo the aura of logical inevitability.

.. Google Ventures, for example, is seriously funding research into immortality. Their head VC will call you a “deathist” for pointing out that this is delusional.

.. Those who benefit from the death of privacy attempt to frame our subjugation in terms of freedom, just like early factory owners talked about the sanctity of contract law. They insisted that a worker should have the right to agree to anything, from sixteen-hour days to unsafe working conditions, as if factory owners and workers were on an equal footing.

.. Many of you had to obtain a US visa to attend this conference. The customs service announced yesterday it wants to start asking people for their social media profiles. Imagine trying to attend your next conference without a LinkedIn profile, and explaining to the American authorities why you are so suspiciously off the grid.

..  All of the major players in the surveillance economy cooperate with their own country’s intelligence agencies, and are spied on (very effectively) by all the others.

.. Try to imagine this policy enacted using the tools of modern technology. The FBI would subpoena Facebook for information on every user born abroad. Email and phone conversations would be monitored to check for the use of Arabic or Spanish, and sentiment analysis applied to see if the participants sounded “nervous”. Social networks, phone metadata, and cell phone tracking would lead police to nests of hiding immigrants.

We could do a really good job deporting people if we put our minds to it.

.. That this toolchain for eliminating enemies of the state is only allowed to operate in poor, remote places is a comfort to those of us who live elsewhere, but you can imagine scenarios where a mass panic would broaden its scope.

.. Or imagine what the British surveillance state, already the worst in Europe, is going to look like in two years, when it’s no longer bound by the protections of European law, and economic crisis has driven the country further into xenophobia.

.. Or take an example from my home country, Poland. Abortion has been illegal in Poland for some time, but the governing party wants to tighten restrictions on abortion by investigating every miscarriage as a potential crime. Women will basically be murder suspects if they lose their baby. Imagine government agents combing your Twitter account, fitness tracker logs, credit card receipts and private communications for signs of potential pregnancy, with the results reported to the police to proactively protect your unborn baby.

.. When we talk about the moral economy of tech, we must confront the fact that we have created a powerful tool of social control. Those who run the surveillance apparatus understand its capabilities in a way the average citizen does not. My greatest fear is seeing the full might of the surveillance apparatus unleashed against a despised minority, in a democratic country.

What we’ve done as technologists is leave a loaded gun lying around, in the hopes that no one will ever pick it up and use it.

..  I am very suspicious of attempts to change the world that can’t first work on a local scale. If after decades we can’t improve quality of life in places where the tech élite actually lives, why would we possibly make life better anywhere else?

.. We should not listen to people who promise to make Mars safe for human habitation, until we have seen them make Oakland safe for human habitation.

.. The goal should be not to make the apparatus of surveillance politically accountable (though that is a great goal), but to dismantle it.

.. This is not the first time an enthusiastic group of nerds has decided to treat the rest of the world as a science experiment. Earlier attempts to create a rationalist Utopia failed for interesting reasons, and since we bought those lessons at a great price, it would be a shame not to learn them.

There is also prior art in attempts at achieving immortality, limitless wealth, and Galactic domination. We even know what happens if you try to keep dossiers on an entire country.

If we’re going to try all these things again, let’s at least learn from our past, so we can fail in interesting new ways, instead of failing in the same exasperating ways as last time.