Should we think of Big Tech as Big Brother?

Many adjectives could be used to describe Shoshana Zuboff’s latest book: groundbreaking, magisterial, alarming, alarmist, preposterous. One will do: unmissable.

As we grope around in the darkness trying to grasp the contours of our digital era, The Age of Surveillance Capitalism shines a searing light on how this latest revolution is transforming our economy, politics, society — and lives. As the inventor of the concept of “surveillance capitalism”, Zuboff fulfils a persuasive role in explaining the ways in which this “voracious and utterly novel commercial project” is radically rewriting the rules of the economic game, creating extraordinary new asymmetries of knowledge and power. By tracking our every click, our every digital expression of interest, ambition, longing and desire, the surveillance capitalists can climb inside our heads and sell those behavioural insights on to their real customers, the advertisers.

Although Zuboff is a professor emerita at the Harvard Business School, her account is as much polemical as it is academic. She is determined to awaken our sense of astonishment and outrage about how this rogue capitalism has evolved to dominate and degrade our lives, largely unnoticed and unchallenged. She also makes one other more worrying and highly contentious claim: that surveillance capitalism has created a new kind of unaccountable power: instrumentarianism. She defines this as the instrumentalisation of behaviour for the purposes of modification, prediction, monetisation and control that threatens to challenge some of the functions of the state and usurp the sovereignty of the people. Instrumentarianism can determine the ends, because it can manipulate the means.

Because this is an unprecedented development, we fail to understand what is happening, just as we failed to comprehend the rise of totalitarianism in 1920s Europe. Moreover, even if we sound the alarm, we should not expect governments to arrest these pernicious developments because they are in many ways their complicit beneficiaries. The security dictates of the post-9/11 world meant that US administrations have given a largely free pass to the tech companies to develop surveillance capitalism, while the Chinese leadership has jumped on it as a powerful means of entrenching control.

The main focus of Zuboff’s analysis, and the primary target of her attack, is Google. She argues that the search company invented and perfected surveillance capitalism in much the same way as General Motors invented and perfected managerial capitalism a century ago. But other practitioners of surveillance capitalism come in for some vicious pummeling too, most notably Facebook and Microsoft.

In Zuboff’s view, Google’s original mission of making all the world’s information accessible mutated into a ruthless imperative to make money by exploiting and modifying human behaviour, by serving up ads to users just at the moment they are the most susceptible to persuasion and generating wants they did not know they had.

Google is now pursuing a land grab for all our data, inventing new products to vacuum up every morsel on the digital map. Every “smart” device — from digital assistants, to rectal thermometers, self-driving cars, to connected homes — has become a data-gathering mechanism. They all serve as one-way mirrors allowing the surveillance capitalists to spy on us without us ever seeing what is going on behind the glass. “There was a time when you searched Google, but now Google searches you,” she writes.

Seven of Google’s products and platforms engage 1bn active monthly users: Gmail, Android, Chrome, Maps, Search, YouTube and Google Play Store, enabling the company to track ever more areas of a user’s life. Similarly, Facebook has more than 2bn users and is also expanding its interests into the physical world. She compares the expansionism of the surveillance capitalists to that of the Spanish conquistadors, who staked claims to virgin territories in the New World while suppressing the unsuspecting native Americans.

Yet there is a two-sided nature to surveillance capitalism that makes it so dangerous, in Zuboff’s view, concealing the dark reality behind the public illusion. Google’s users are not its customers, which means it is radically indifferent to their real interests. Advertising-supported search engines will always prioritise those who pay the bills over those who use its services, so long as they remain hooked.

That also used to be the view of Sergey Brin and Larry Page, Google’s founders, who presented a paper in 1998 highlighting the perils of advertising. “We expect that advertising-funded search engines will be inherently biased towards the advertisers and away from the needs of consumers. This type of bias is very difficult to detect but could still have a significant effect on the market,” they wrote.

The expansionism of the surveillance capitalists is like that of the Spanish conquistadors, who staked claims to virgin territories in the New World

That worldview changed when Google realised that the behavioural insights it could draw from its data were a potential gold mine, offering far greater rewards than other advertising-driven businesses such as commercial television. Zuboff argues that the clearest logic of surveillance capitalism has been articulated by Hal Varian, Google’s chief economist, who is sometimes referred to as the “godfather” of the company’s advertising model. As an expert in computer-mediated transactions, Varian identified four features of this new economy: data extraction and analysis; new contractual forms due to better monetisation; personalisation and customisation; and continuous experimentation. It was that formula that enabled Google to generate its vast wealth and has been copied by so many others since.

What makes surveillance capitalism particularly pervasive is the way it preys upon our behavioural vulnerabilities. Indeed, Zuboff argues that the ability to trade in our behaviour has created a new “commodity fiction”, to use Karl Polanyi’s classification. In this historian’s view, the market economy only really took off when three mental “inventions” developed, in the form of labour, land and money, turning abstract concepts into exchangeable commodities. In a similar way, Zuboff says, the surveillance capitalists have been able to commoditise the fiction of behaviour, turning our data into profit. “They no longer merely host content but aggressively, secretly, and unilaterally extract value from that content.”

Where Zuboff strays into still more controversial ground is in her analysis of power. Surveillance capitalists are not only able to monetise our data but can also use it to predict our behaviour and thereby modify it. In mechanical terms, they are no longer just sensors but actuators.

She traces a straight line from the controversial behaviourist theories of BF Skinner, the Harvard psychologist from the 1960s, to those espoused today by Alex Pentland, the MIT professor and author of Social Physics. In their view, she claims, computational “truth” might prove a more effective way of running society than human governance. The intent is to replace our political autonomy with heteronomy, or the rule of others. But who will determine the values of this Big Other?

China’s vision is to use technology to monitor its citizens and assign them social credit scores, used to reward or punish citizens for what the authorities deem to be socially good or bad behaviour. Having destroyed trust by annihilating all social institutions, the Communist party has seen technology as a means of recreating it artificially. This is best understood as the “apotheosis of instrumentarian power fed by public and private data sources and controlled by an authoritarian state.” In the west, Zuboff argues that we are in danger of experiencing an anti-democratic coup des gens rather than a classic coup d’état, an overthrow of the people rather than of the state.

Her grand thesis can be contested on many levels. She largely ignores the positive side of our technological revolution. She almost certainly understates the competitive dynamics of the market. And she portrays the young as helpless saps, who use their phones 157 times a day, even as they appear to be becoming ever more savvy about and sceptical of technology.

Zuboff’s analysis of power is also debatable. After all, it is a strange kind of power that can be wiped off a phone in a matter of seconds. Besides, the people who theoretically wield power appear to have little interest in exercising it, beyond enriching themselves and their shareholders. They have no grand project for humanity other than vague notions of doing good. The cardinal sin of the likes of Larry Page, Mark Zuckerberg and Satya Nadella is surely not that they are evil but that they are naive.

Extrapolation of current trends also leads to some extravagant conclusions. The evidence for suggesting that surveillance capitalism may be increasing control over human actions is somewhat sketchy, to say the least. In the west, at least, the greater danger may be Facebook-inspired anarchy, rather than excessive control, as protest movements like the gilets jaunes show in France.

The weakest part of The Age of Surveillance Capitalism, though, concerns what she herself would do about the concerns she raises. If the problems are so urgent then why are her proposed responses not more radical? She suggests that 20th-century conceptions of privacy laws and antitrust policies are outdated, but sketches out few ways in which they could be updated.

She says the debate should revolve around three questions: Who Knows? (who has access to knowledge); Who Decides? (which institutions have authority); And Who Decides Who Decides? (a question of power in deciding which institutions have authority). Having raised so many stark questions, her book tends to peter out in a series of mumbled answers.

But her conclusions are surely right on at least two fronts.

First, she attacks technology’s ideology of inevitabilism (even though that tends to undermine her case about its growing omniscience). Just because technology has turned out the way it has does not mean that was the only way the internet could have evolved. Its development was contingent on circumstance and influenced by individuals. Societies can still choose to use technology in different ways so long as they can mobilise sufficient action.

Second, we should be constantly wary of those offering sweeping technological solutions to human problems. Such “solutionist” remedies are invariably worse than the original disease.

Robert Conquest, the late conservative historian, once argued that many of the greatest tragedies of the 20th century stemmed from “solutions” rather than problems. Eliminating undesirable classes or races, as in Stalin’s Soviet Union, Hitler’s Germany, or Mao’s China, were monstrously worse outcomes than any of the problems they were supposed to “solve”.

It is always better to grapple with problems in imperfect, messy, human ways than to jump to all-embracing, inhuman solutions, no matter how decisive or alluring they may seem to some.

The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power, by Shoshana Zuboff, Profile Books, RRP£25, 692 pages

John Thornhill is the FT’s innovation editor

Join our online book group on Facebook at FTBooksCafe. Subscribe to FT Life on YouTube for the latest FT Weekend videos

Letter in response to this article:

China is nudging us into a world governed by data / From Marc Sylvain, Santa Barbara, CA, US