In 2017, a team led by Sandra Matz ran a set of field experiments on Facebook with a scale that is hard to picture. Working with more than three and a half million people, the researchers matched advertising copy to the personality profiles the platform had already inferred from people’s Likes. Ads tuned to a person’s extraversion or their openness to experience drew up to 40 percent more clicks and up to 50 percent more purchases than untuned versions of the same pitch. The mechanism was unremarkable: people respond more warmly to language that fits them. What was new was the automation. Nobody had to read the room. A model read it instead, for millions of rooms at once.
That study is a useful place to begin because it is often cited as proof of something far grander than it demonstrated. It shows that personality-matched persuasion beats generic persuasion in a large online population. It does not show that anyone can reprogram a person’s beliefs, and it does not show that the technique works the same way on everyone or in every domain. The gap between those two claims is where the real argument lives. The question this article takes up is narrow and urgent: what happens when personalization is joined to continuous measurement of a person’s internal state, so that the pitch can be rewritten in real time?
The three capabilities that combine
It helps to separate what is already routine, what is plausible engineering, and what remains speculation. Contemporary systems already have the first capability, personalization, in industrial form. Recommender engines tailor what appears in a feed to inferred preferences, and Matz’s experiments show that the same tailoring extends to the wording of a persuasive appeal. The second capability, measurement, is also mature in a thin sense: platforms track clicks, dwell time, scrolling, purchases, and time of day. They infer emotional and motivational states from those traces. The third capability, adaptation, ties the two together. A static campaign gives everyone the same message and never learns. An adaptive system treats each response as feedback, estimates whether the message landed, and adjusts.
The important insight is that these three capabilities are not additive. They form a loop, and a loop can compound in ways that a single well-crafted message cannot. A persuasive essay is a one-time intervention. A persuasive system that measures the effect of its own intervention and updates is a different kind of object: it is a control system whose target is a human being. This is what makes the phrase “manipulation machine” more than rhetoric. The machine is not a message. It is a feedback circuit.
What the evidence for personalized persuasion actually shows
The research base is stronger than skeptics admit, and weaker than alarmists claim. Consider three studies at increasing levels of automation.
Matz and colleagues established the basic effect in 2017: personality-matched ads outperformed mismatched ones, and the platform infrastructure needed to run the experiment at scale already existed. That is demonstrated science.
In 2023, a team led by Ben Tappin at MIT examined political microtargeting directly. In their first study, messages tailored to individual voters’ psychological profiles outperformed several alternatives by a wide margin, in the range of seventy percent or more. But the same paper contains a crucial corrective. Targeting on more than one psychological trait at a time produced no additional benefit over targeting on a single trait, and the advantage shrank or vanished in other conditions. The authors’ own framing was cautious: microtargeting helps, sometimes substantially, but it is not mind control, and its returns depend heavily on context. Anyone who cites this paper as proof that campaigns can push any voter in any direction has not read it.
The most consequential recent work pushes the automation further. In 2024, Matz and colleagues published a series of seven sub-studies in Scientific Reports, with a combined sample of 1,788 participants, testing messages written with the help of a large language model. The AI-personalized messages were more persuasive than non-personalized ones across multiple domains, and, notably, telling people that a machine had written the message did not reduce its effect. That last finding matters for regulation. Transparency about AI authorship is ethically necessary, but it may not blunt the influence.
A parallel result from the same year measured how far automated text generation could go on its own. Josh Goldstein, Jason Chao, Shelby Grossman, Alex Stamos, and Michael Tomz compared human-written propaganda, GPT-3-written propaganda, and a combined effort across 8,221 US respondents. Agreement with the control baseline ran at about 24 percent. Human-written propaganda moved agreement to about 47 percent; AI-written propaganda reached about 43.5 percent. When humans and the model worked together, the figure climbed past 50 percent, at least matching the human authors. Machine-generated persuasion is not a promise. It is a measured competitor to skilled human propagandists.
Together these studies describe a capability that is real, generalizable, and improving. They also describe something bounded. Personalized persuasion shifts probabilities and opinions at the margin. It does not dissolve a person’s judgment. The honest reading is that the machine is powerful in the way a very good sales team is powerful, and that this comparison should not comfort anyone, because very good sales teams have reshaped markets, elections, and lives.
From persuasion to a closed loop
The step that changes the character of the technology is the closing of the loop, and here the evidence becomes thinner and more indirect. The most-cited experiment is Adam Kramer, Jamie Guillory, and Jeffrey Hancock’s 2014 study on emotional contagion, which manipulated the emotional content of the Facebook News Feed for 689,003 users and found that reducing positive emotional content shifted users’ subsequent posting toward the negative, and vice versa. The finding is important and frequently misused. It demonstrates that a platform-scale manipulation of what people see can shift their expressed emotional state, which is precisely the measurement-and-intervention loop in miniature. It should be cited with its caveat: the paper carries a formal Expression of Concern from PNAS over the adequacy of informed consent, and the ethical objection it generated is itself part of the lesson.
That experiment measures the effect of the platform on the user. A true manipulation machine would also run in the other direction, inferring the user’s state and feeding it back into the next intervention. Some of that already happens without any neuroscience. An ad auction that bids differently based on time of day, device, past behavior, and inferred mood is a rudimentary version. A recommender that notices a user binging on outrage content and serves more is another. What does not yet exist at consumer scale is a system that reads a person’s physiology or neural activity and adapts a persuasive intervention in the same second. That is the frontier the phrase “most powerful manipulation machine ever built” is really pointing at, and it is important to say plainly that this frontier is engineering speculation, not a shipped product.
The reason to take the speculation seriously anyway is that its components are all under active development. Wearables already estimate heart rate variability, sleep, and stress. Research systems infer cognitive and emotional states from language, voice, and gaze. Interfaces that stimulate or read neural activity exist in clinical and experimental settings. If state inference becomes accurate and cheap, then the missing piece for a closed loop is only the willingness to connect it to a persuasive objective. Nothing in the technology forbids that connection. Only law, ethics, and business incentives stand in the way, which is exactly why those things deserve attention now rather than after the fact.
The cautionary tale that should be told accurately
No discussion of this subject escapes Cambridge Analytica, and no discussion should, provided it is told accurately. The firm’s public reputation rests on a claim it could not support: that it had built psychological profiles detailed enough to manipulate voters individually. The Federal Trade Commission’s 2019 administrative action tells a more modest and more instructive story. The Commission found that Cambridge Analytica deceived consumers about the collection of their personal information, gathering data from roughly 250,000 to 270,000 Facebook users through a quiz app and reaching tens of millions of their friends, while falsely asserting that it did not collect identifying information. In December 2019 the FTC issued a final order settling the charges and barring the firm from making such misrepresentations.
Notice what the record supports and what it does not. It supports a finding of deception, a real violation of the FTC Act, and a real failure of platform governance. It does not support the story that a secret algorithm swayed an election by reaching into individual minds. The most important artifact of the Cambridge Analytica affair is not a manipulation technique. It is the demonstration that a company can market an exaggerated manipulation capability and that its customers and the public will believe it. The myth of the machine is itself a tool of influence, because a target who believes they are being manipulated is easier to demoralize.
The machinery that is hiding in plain sight
The durable manipulation infrastructure is less cinematic than Cambridge Analytica and more effective: ranking and recommendation. In 2022, Ferenc Huszár and colleagues published a study in PNAS that assessed how Twitter’s algorithmic timeline amplified political content. By comparing a randomized control group that kept the reverse-chronological feed against users exposed to the algorithmic feed, across roughly two million accounts and seven countries, they found that the algorithm amplified mainstream right-leaning content more than mainstream left-leaning content in six of the seven countries, while finding no evidence that far-left or far-right content was amplified more than moderate content. Two things follow. First, ranking algorithms demonstrably alter the political information environment at population scale; that is demonstrated science. Second, the direction of the effect is conditional and platform-specific, which is why loose talk about algorithms uniformly favoring “one side” tends to be wrong.
What the ranking evidence and the Cambridge Analytica record have in common is a pattern of influence that operates through the environment rather than through argument. It does not command belief. It arranges circumstances so that some actions become frictionless and others exhausting. That is a form of power that requires no theory of mind at all, and much of it is already here.
What a real closed loop would require
It is worth spelling out the engineering honestly, because the honesty clarifies where the risk actually sits. A closed-loop persuasion system needs four things.
It needs a sensor. Today, behavior is the dominant sensor. Physiology and neural signals are secondary and, outside clinical settings, unreliable. The hard problem is not attaching a device. It is establishing that any given signal means what the system claims it means.
It needs an inference layer that is honest about uncertainty. A pulse is not fear. A pause is not deception. A pattern of brain activity is not a verdict on a person’s values. Good systems carry uncertainty forward and preserve competing explanations. Bad systems collapse a signal into a label and act on it with confidence it has not earned.
It needs an intervention that can be delivered and varied at scale. Personalized text generation already meets that bar, as the 2024 studies show. Physical interventions, from medication to neural stimulation, do not, and treating them as equivalent is a category error.
It needs an objective function, and this is the crux. A loop that serves a person’s considered goals is a therapeutic tool. A loop that serves an advertiser, an employer, a political movement, or an engagement metric is something else entirely, even if the software is identical. The moral character of the machine is set by whose goal sits in the driver’s seat. This is why debates about “is the AI good or bad” miss the point. The question is always: optimizing for whom, and toward what?
Consent and the problem of the moving baseline
Most ethical frameworks for these systems lean on consent, and consent is necessary but plainly insufficient in a closed loop. A worker may click “agree” because the alternative is losing a job. A user in an emotionally altered state may consent to continue the very intervention that produced the altered state, and their preference at that moment may not reflect what they would choose calm. A platform may disclose everything truthfully while designing the service so that dependence is the economically rational outcome. Informed consent assumes a stable chooser. A manipulation loop specifically targets the chooser’s state.
A more defensible principle is baseline control. Significant changes to how a person is treated should be reversible, inspectable, and capable of being reaffirmed at a later time when the person is not under the influence of the intervention in question. The analogy is not a privacy policy but a clinical trial: you check in with the patient after the drug wears off, not only while it is active. Applied to software, this means preserving a record of what the system did, allowing the user to see and reverse it, and treating the ability to say no after the fact as more important than the ability to say yes in the moment.
What would actually change the argument
Debate on this subject produces more heat than evidence, so it is worth being specific about what would move the question.
First, independent, replicated demonstrations. A single viral study of emotional contagion or microtargeting cannot carry the weight that public anxiety places on it. The field needs interventions that reproduce across platforms, populations, and time, and it needs null results to be published alongside the positive ones.
Second, results that transfer. A manipulation that works only in one laboratory, on one topic, using one interface is not the machine people fear. The interesting and worrying question is whether effects survive when the context changes, because that is what would turn a trick into a durable capability.
Third, evidence beyond self-report. If a system changes behavior while users report that their preferences are unchanged, that asymmetry is the signature of the kind of influence that should concern us. If, instead, users can consistently detect, resist, and reverse the intervention, the machine is less powerful than the myths suggest.
Fourth, regulation and product architecture that separate state inference from persuasion objectives. The clearest structural safeguard is to prevent the same system that reads your state from also being the system that profits from changing it. The EU’s Digital Services Act points in this direction: Article 27 requires platforms to disclose the main parameters of their recommender systems, Article 38 requires very large platforms to offer a non-profiling alternative to their recommender systems, and Article 25 prohibits interfaces that deceive or manipulate users. These are early, imperfect levers. Their value is that they treat ranking and interface design as objects of governance rather than as private and neutral choices.
The larger stake
The reason this topic resists easy resolution is that it sits at the intersection of several disciplines that rarely talk to each other. Engineers see a control problem. Economists see an incentive problem. Lawyers see a consent problem. Theologians and ethicists see a problem about what a person is for, and whether a human being may be treated as an input to someone else’s optimization. All of them are right about part of it.
A system can be extraordinarily intelligent and still pursue the wrong end with extraordinary efficiency. That is the real danger. The most powerful manipulation machine is not dangerous because it is clever. It is dangerous because a clever system aimed at the wrong objective will find, faster and more cheaply than any human operator, the precise set of circumstances under which a person acts against their own considered interest. The boundary that matters most is the one that keeps intelligence in an advisory role: a system may help a person clarify their goals, but it should not thereby acquire the authority to choose those goals for them. The more intimate the sensing and the more personal the persuasion, the more that boundary has to be built, defended, and enforced.
Sources and further reading
- Matz, S. C., Kosinski, M., Nave, G., & Stillwell, D. J. (2017). “Psychological targeting as an effective approach to digital mass persuasion.” Proceedings of the National Academy of Sciences, 114(48), 12714–12719. https://www.pnas.org/doi/10.1073/pnas.1710966114
- Tappin, B. M., Wittenberg, C., Hewitt, L. B., Berinsky, A. J., & Rand, D. G. (2023). “Quantifying the potential persuasive returns to political microtargeting.” Proceedings of the National Academy of Sciences, 120(25), e2216261120. https://www.pnas.org/doi/10.1073/pnas.2216261120
- Matz, S. C., Teeny, J. D., Vaid, S. S., Peters, H., Harari, G. M., & Cerf, M. (2024). “The potential of generative AI for personalized persuasion at scale.” Scientific Reports, 14, 4692. https://www.nature.com/articles/s41598-024-53755-0
- Goldstein, J. A., Chao, J., Grossman, S., Stamos, A., & Tomz, M. (2024). “How persuasive is AI-generated propaganda?” PNAS Nexus, 3(2), pgae034. https://academic.oup.com/pnasnexus/article/3/2/pgae034/7610937
- Kramer, A. D. I., Guillory, J. E., & Hancock, J. T. (2014). “Experimental evidence of massive-scale emotional contagion through social networks.” Proceedings of the National Academy of Sciences, 111(24), 8788–8790. (See the accompanying Expression of Concern: https://www.pnas.org/doi/10.1073/pnas.1412469111) https://www.pnas.org/doi/10.1073/pnas.1320040111
- Huszár, F., Ktena, S. I., O’Brien, C., Belli, L., Schlaikjer, A., & Hardt, M. (2022). “Algorithmic amplification of politics on Twitter.” Proceedings of the National Academy of Sciences, 119(1), e2025334119. https://www.pnas.org/doi/10.1073/pnas.2025334119
- Federal Trade Commission. (2019). “FTC Issues Opinion and Order Against Cambridge Analytica.” https://www.ftc.gov/news-events/news/press-releases/2019/12/ftc-issues-opinion-order-against-cambridge-analytica-deceiving-consumers-about-collection-facebook
- European Union. (2022). Regulation (EU) 2022/2065 (Digital Services Act), Articles 25, 27, and 38. https://eur-lex.europa.eu/eli/reg/2022/2065/oj/eng
Loading comments…