In 1958 Aldous Huxley graded his own prophecy against George Orwell’s and found it more plausible. “It now looks as though the odds were more in favor of something like Brave New World than of something like 1984,” he wrote in Brave New World Revisited. The line is usually quoted for atmosphere. It reads better as a mechanism claim, and claims about mechanism can be examined.
Comfort is cheaper than coercion
Huxley’s argument was not that tyranny would turn kind. It was that terror is an inefficient instrument, and that a society run on reinforcement would outlast one run on fear.
“Control through the punishment of undesirable behavior is less effective, in the long run, than control through the reinforcement of desirable behavior by rewards,” he wrote. He pushed the same reasoning further: “government through terror works on the whole less well than government through the non-violent manipulation of the environment and of the thoughts and feelings of individual men, women and children.”
Parts of that have not aged well. Behaviorist confidence about reinforcement collided with the overjustification effect: paying people for what they already enjoy can corrode the internal reason they did it, and organizations that try to reward everything learn this the expensive way. The modest version of Huxley’s claim survived the wreckage, though. Given two durable ways to make a population behave, the cheaper one is usually to arrange the environment so the preferred behavior becomes the path of least resistance and then let people choose it.
Neil Postman converted this into cultural criticism in Amusing Ourselves to Death (1985). Orwell imagined a public crushed by state violence; Huxley imagined one “oppressed by their addiction to amusement,” medicating itself into bliss and trading away its rights in the process. Postman’s medium was television, and his thesis was that a medium makes some kinds of thought easy and others laborious, until taste drifts to match. Replace broadcast with ranked feeds and the logic barely changes.
The distinction matters because it changes what a warning is for. If the danger is a boot on a neck, the remedy is a right, a court, and a lawyer. If the danger is a default setting, the remedy looks like product architecture, procurement language, and employment law — dull levers, but usually the ones that decide outcomes.
The strongest experimental case for quiet influence
The most cited demonstration that ranking can move political preference without detection is the search engine manipulation effect, reported by Robert Epstein and Ronald Robertson in PNAS in 2015. Their five double-blind randomized experiments involved 4,556 undecided voters in the United States and India, including a study run across India during the 2014 Lok Sabha elections. Biased search rankings shifted the voting preferences of undecided voters by twenty percent or more, with larger shifts in some demographic groups, and the participants showed no awareness that they had been manipulated.
Two qualifications belong next to that result. Google publicly disputed the interpretation, and the measurement concerned a shift in stated preference inside an experiment, not a demonstrated reversal of a real election outcome. The paper’s own framing is conditional: knowing how many undecided voters a population contains, and how many of them are susceptible, yields a threshold below which a search ranking order could plausibly decide a close contest.
A second strand of work shows the same effect through social proof rather than ranking. Robert Bond and colleagues, publishing in Nature in 2012, ran a randomized controlled trial of political mobilization messages delivered to 61 million Facebook users during the 2010 United States congressional elections. The messages influenced political self-expression, information seeking, and verified real-world voting. Transmission through close friends produced a larger effect on real-world voting than the direct messages did, and nearly all of that transmission occurred between ties who plausibly saw each other in person.
Read together, these experiments establish something narrow and important: attention-ordering and social cues can change what people do, at scale, while leaving them feeling that nothing in particular happened to them. That is the mechanism Huxley described, demonstrated in miniature rather than assumed.
The counter-evidence is just as real
The political persuasion literature cuts hard against the strong version of the Huxley story. Large, well-powered field experiments keep finding that influence at scale is small, context-dependent, and quick to decay.
Minali Aggarwal and colleagues reported in Nature Human Behaviour in 2023 on an $8.9 million campaign-wide experiment delivered to two million persuadable voters across five battleground states during the 2020 United States presidential election. The program produced no detectable change in turnout on average. The only positive finding was a small differential effect: voting rose among voters modelled as leaning Biden by 0.4 percentage points and fell among voters modelled as leaning Trump by 0.3 points.
Hunt Allcott, Matthew Gentzkow and colleagues went further in Nature Human Behaviour in 2025. They randomized 36,906 Facebook users and 25,925 Instagram users to have political advertising removed from their feeds for the six weeks before the 2020 election. Removing political ads produced no detectable effects on political knowledge, affective polarization, perceived legitimacy of the election, participation including campaign contributions, candidate favorability, or turnout — overall or by party.
The microtargeting literature is similarly sobering. Ben Tappin and colleagues, writing in PNAS in 2023, did find a microtargeting advantage of seventy percent or more over alternative strategies in one advocacy context. The advantage disappeared when targeting on more than one covariate, proved limited in a second study, and depended heavily on circumstances. Alexander Coppock, Donald Green and Ethan Porter, reporting in Research & Politics in 2022, randomized Facebook and Instagram advertising across Florida ZIP codes during the 2018 midterms: more than 1.1 million impressions produced an estimated effect on Democratic vote share of −0.04 percentage points with a standard error of 0.85.
The honest synthesis is that engineered influence works best where attention is captive and the comparison set is narrow, and works poorly where audiences are large, distracted, and exposed to competing messages. A search results page with one dominant provider is closer to the first condition. A noisy national media environment is closer to the second. That difference constrains the Huxley scenario, and it also tells you where to look for the conditions that would make it real.
Where influence gets cheap
The persuasion results are not contradictory once you ask what makes a message cheap to deliver. Two variables do most of the work: how captive the attention is, and how narrow the choice set is.
Search rankings sit at the favorable end of both. The user has already stated an interest, the results arrive in an order the user cannot see the reasons for, and there is usually only one interface in front of them. That is why the manipulation effect was detectable there and why the same effect has been hard to reproduce in contexts where people are interrupted, distracted, and told a dozen contradictory things a day. A campaign ad competes with the rest of a life. A ranked list of answers is the whole of the interaction.
Social proof occupies a middle position. The 61-million-person experiment worked because the cue came from people the recipient actually knew, not because Facebook had argued anyone into anything. The message borrowed the authority of a close tie rather than manufacturing a preference from nothing. That is a real and repeatable mechanism, and it is also bounded: it scales with the density of the underlying social graph, which is not something an operator can conjure.
The practical implication is uncomfortable. The places where soft influence is strongest are the places people depend on most and can least easily leave: the search engine everyone uses, the messaging app the whole contact list lives on, the employer’s scheduling system, the state’s benefits portal. Influence is cheap exactly where exit is expensive. That correlation is the heart of the Huxley worry, and it does not require any exotic technology to hold.
The design question hiding inside the alarm
Critics of soft control often slide from “this mechanism works” to “therefore it is being used against you.” The slide is not licensed, and the fix is to ask a narrower question that has an answer: who chooses the objective, and who bears the cost of the choice?
Consider a safety system that detects a drowsy driver. It reads a body state the driver did not volunteer, and it does so to keep the driver and the road alive. Regulators treat this kind of monitoring differently for a reason. The European Union’s AI Act excludes physical states such as pain and fatigue from its definition of emotion recognition, explicitly citing systems that detect the fatigue of professional pilots or drivers to prevent accidents. The same statute bans AI systems used to infer emotions in the workplace — unless the use is strictly for medical or safety reasons. The carve-out is narrow on purpose, and the Commission’s guidelines read it that way: detecting tiredness to prevent a crash is inside the exception; inferring engagement, boredom, or job satisfaction is not.
Now run the same system with a different objective. The sensor does not change; the person reading the output does. Sleep-deprived workers become visible as a compliance problem, then as a hiring filter, then as a term in a productivity score. Nothing about the technology announced this. It arrived through a sequence of individually reasonable decisions, each made by someone whose incentives pointed the same direction.
That is the pattern worth naming, and it is more specific than a general fear of surveillance. The failure is not that a machine can infer a state. The failure is a mismatch between who pays for the sensing and who benefits from it. When the person wearing the sensor gets the information and decides what to do with it, the same measurement that would otherwise be coercive becomes protective. When the employer or the agency holds it, the measurement is evidence, and evidence accumulates into judgment.
This is why the design question is harder than the ethics question. Almost any soft-control mechanism can be described in a way that makes it sound benign, because the underlying operation — observe, infer, adjust — is exactly what a helpful system does. The distinction that survives scrutiny is not about the technique. It is about whether the person being measured can see the measure, dispute it, and keep it from being turned into a decision about them.
From prediction to modification
Shoshana Zuboff’s account of surveillance capitalism supplies the vocabulary for the next step. Her 2015 article in the Journal of Information Technology describes a new logic of accumulation built on the unilateral claiming of behavioral data as raw material, converting it into predictions that can be sold. She names the resulting power structure “Big Other,” and argues that its mechanisms of extraction and control tend to exile people from their own behavior while creating markets for behavioral prediction and modification.
That analysis is interpretive social theory, not measurement, and it should be read as such. Its useful contribution here is a distinction that empirical work often blurs: predicting what someone will do differs from acting to change what they do. Most deployed systems perform the first and produce the second only as a side effect of incentives. Netflix does not need to make viewers prefer anything; it needs to place an option where it will be taken. A feed does not need to persuade; it only needs to keep the next scroll cheap.
The distinction matters for how alarmed to be. Prediction without modification produces filter bubbles, unequal access, and price discrimination. Modification without consent produces something closer to what Huxley feared. The regulatory signal is worth reading carefully here, because regulators have begun writing rules that assume modification is possible. The European Union’s AI Act prohibits AI systems that deploy subliminal or purposefully manipulative techniques to materially distort behavior in ways that cause significant harm, and separately bans social scoring by public authorities. Those prohibitions exist because legislatures judged the mechanism credible enough to be worth banning in advance.
A three-way ledger: measured, buildable, speculative
Keeping three categories separate prevents most of the confusion in this debate.
Demonstrated. Randomized trials show that ranking order, social proof, and default placement shift attitudes, purchases, and votes by measurable amounts. Effects on stated preference can exceed twenty percent in controlled settings; effects on aggregate real-world behavior in large multiparty environments are typically under one percentage point and often indistinguishable from zero.
Plausible engineering. Systems that infer internal state from behavioral traces and adapt their presentation accordingly are already deployed at consumer scale, and the same techniques work inside employers’ and states’ administrative software. The AI Act’s high-risk classification for employment, education, and essential-services uses reflects a judgment that these systems warrant documentation, human oversight, and logging — a posture consistent with real capability and real limits.
Speculative. A state that could reliably steer a whole population’s preferences without coercion, without detection, and without paying any cost in legitimacy does not have a supporting evidence base. Nothing in the experiments above shows control that durable. What they show is drift: preferences moving a little, many times, in directions chosen by whoever sets the ordering.
The speculative version is where most public argument happens, and it is the least supported. The demonstrated version is quiet enough that it rarely makes headlines.
Why comfortable control is hard to refuse
Coercion has an advantage that soft control lacks: it is legible, and legibility is the beginning of resistance. A soft system works by making refusal expensive in small, distributed ways.
Consider the mechanics of exit. A user can technically leave a platform, but leaving means abandoning the social graph, the professional network, the messaging history, the searchable public record of their work. A worker can technically decline monitoring, but declining shifts them out of the group whose performance is legible to people who make promotion decisions. Multiply the choice across millions of people and the aggregate effect is comportment: people learn which behaviors are rewarded and adjust, mostly without experiencing the adjustment as a decision.
Postman’s television audience and Huxley’s soma both describe the same equilibrium. Nobody has to be threatened for a population to settle into a pattern, as long as each individual’s path of least resistance points the same way. Huxley’s own summary of the risk was stark: “the nightmare of total organization, which I had situated in the seventh century After Ford, has emerged from the safe, remote future and is now awaiting us, just around the next corner.”
There is a real counterargument, and it deserves a hearing. Comfortable order is often genuinely preferable to the alternatives it displaces. Public health messaging that makes vaccination the default saves lives. Frictionless tax filing increases compliance. Safety systems that detect a drowsy pilot are not instruments of tyranny, even though they read a body state without being asked. The critique of soft control only earns its force if it can distinguish the instances where defaults serve the person following them from the instances where defaults serve the institution setting them. Usually the test is who can change the default, and what happens to the person who opts out.
What would change the picture
Two kinds of evidence would move this argument, and both are findable.
The first is evidence of durable, large, non-coercive preference change in open environments. The field has spent a decade looking for it with good instruments and mostly found effects that are small, conditional, and perishable. If a platform or a state could show replicated twenty-point shifts in real elections using ranking alone, the Huxley thesis would need far less hedging. The absence of such findings is the strongest available argument that the soft version of tyranny has real limits.
The second is evidence of structural concentration: a single intermediary controlling most of the attention in a large population, with no meaningful exit and no outside audit of its ordering. Search engine manipulation effects were strongest in exactly that configuration during the original studies. Where competition and transparency exist, influence is weaker and the manipulation is easier to detect. Where they do not, the same measurable effect sits inside an architecture nobody can see.
That is why the useful tests are institutional rather than psychological. Can a researcher audit the ranking? Can a competitor interoperate? Can a user export their data and leave? Can a worker refuse monitoring without losing standing? Can a citizen learn that a decision about them was automated, and get it reviewed? Each of these is boring, specific, and enforceable in a way that “resist manipulation” is not.
Why the soft version is harder to regulate
Hard control leaves a trail. A censorship order has an author, a decree has a signature, a police action has a record. When something goes wrong, there is a document to point at and a person to hold responsible.
Soft control distributes the responsibility until it evaporates. The ranking engineer optimizes engagement because that is the assigned metric. The product manager ships the default because it tested better. The employer buys the dashboard because the vendor’s case study promised a productivity gain. The legislator permits it because no single component looked dangerous at the time it was considered. Each link in that chain is defensible in isolation, and the aggregate is a system that shapes a population’s behavior without anyone having decided to.
This is a governance problem more than a technology problem, and it explains why the regulatory response has been so backward-looking. Rules get written after a harm is legible. The AI Act’s prohibition on manipulative techniques that materially distort behavior is written to catch a mechanism that harms, which means it needs a victim and a demonstrated causal path before it bites. Ranking bias that shifts preferences by a few points in an election nobody notices has neither. Defaults that quietly reorganize a workplace have neither. The mechanisms the Huxley scenario predicts are precisely the ones that leave no evidence of a decision to point to.
The counterexample is instructive. Biometric time-and-attendance systems are intrusive in an obvious way, and they have been regulated in an obvious way. The United Kingdom’s Information Commissioner’s Office ordered Serco Leisure and several associated trusts to stop using facial recognition and fingerprint scanning to clock employees in and out across 38 leisure facilities, affecting the biometric data of more than 2,000 workers, on the grounds that the processing was neither necessary nor proportionate when less intrusive alternatives such as key fobs existed. The regulator’s framing is the useful part: the question was not whether the technology worked but whether there was a cheaper way to achieve the same legitimate goal. That test is portable, and it is not limited to biometrics.
Apply it to a feed ranking, a recommended schedule, or a performance score, and the analysis gets uncomfortable rather than impossible. Was there a less intrusive way to solve the stated problem? Did the person affected have a real alternative? Was the goal stated honestly, or was the stated goal doing work that the real goal could not defend in public? Those are answerable questions. They just require someone to ask them before deployment rather than after a scandal.
Where this leaves the argument
Huxley was probably right about the class of mechanism and probably optimistic about the degree of control it yields. Punishment is expensive and generates resistance; reward is cheap and generates compliance. That asymmetry is real, it is visible in every ranking system and every workplace dashboard, and it explains why twenty-first-century control looks less like a boot and more like a well-designed interface.
But reinforcement is not omnipotence. People are not blank, persuasion at scale is measurably hard, and a population that can compare notes is hard to steer. The dystopia this machinery actually supports is less dramatic than either novelist imagined: a set of institutions that find it easy to keep people occupied, tracked, and gently nudged, and that discover nobody has a strong reason to object because each individual objection looks disproportionate to its object.
Resisting that outcome will not look like a rebellion. It will look like antitrust enforcement, ranking transparency, employment protections, data portability, and the unglamorous work of making sure that the people whose preferences are being shaped have somewhere else to go. Those are the levers that decide whether comfortable control stays a description of advertising and becomes a description of government. The next article in this series takes the mechanism somewhere more concrete, where the asymmetry of power is sharpest: the workplace, where an employer can measure a body and a schedule can be adjusted in response.
Sources and further reading
- Aldous Huxley, Brave New World Revisited (1958), full text: huxley.net/bnw-revisited
- Neil Postman, Amusing Ourselves to Death: Public Discourse in the Age of Show Business (Viking, 1985), overview: Wikipedia
- Robert Epstein and Ronald E. Robertson, “The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections,” PNAS 112(33), 2015: pnas.org
- Robert M. Bond et al., “A 61-million-person experiment in social influence and political mobilization,” Nature 489, 2012: nature.com
- Ben M. Tappin et al., “Quantifying the potential persuasive returns to political microtargeting,” PNAS 120(25), 2023: pnas.org
- Minali Aggarwal et al., “A 2 million-person, campaign-wide field experiment shows how digital advertising affects voter turnout,” Nature Human Behaviour 7, 2023: nature.com
- Hunt Allcott, Matthew Gentzkow et al., “The effects of political advertising on Facebook and Instagram before the 2020 US election,” Nature Human Behaviour, 2025: nature.com
- Shoshana Zuboff, “Big Other: Surveillance Capitalism and the Prospects of an Information Civilization,” Journal of Information Technology 30(1), 2015: hbs.edu
Loading comments…