There is a particular institutional move that is easy to miss because it resembles care. A worker is exhausted, angry, or grieving, and the organization responds by attending to the worker’s state rather than the conditions producing it. The distress is real. The response is also real. What quietly disappears is the possibility that the distress was accurate — a correct reading of a situation that should not have been tolerated in the first place.
Medicine has already run a version of this experiment on itself, and the results are instructive. For years the distress of clinicians was filed under “burnout,” a term that locates the problem inside the individual. In 2019, the psychiatrist Wendy Dean and the surgeon Simon Talbot, writing with Austin Dean, argued that the label was both inaccurate and convenient, and proposed a different one: moral injury. Moral injury describes what happens when people are repeatedly forced, by constraints they do not control, to act against their deepest professional commitments — in this case the oath to put patients first. The distinction they drew is the one this article is about. Burnout implies a deficient person who needs rest or resilience training. Moral injury locates the damage in a system that has made integrity impossible. The second framing points at the institution; the first points at the patient. And the first framing is cheaper, because a burned-out worker can be sent to a wellness seminar, while a morally injured workforce is a signal that the organization itself is failing.
That substitution — calling a legitimate response to a bad situation a disorder of the responder — is the mechanism I want to examine here. It is old. What is new is that machine learning may make it cheap, personal, and continuous. Once a system can infer your internal state, recommend an intervention, and adapt that intervention in real time, the cost of adjusting the person can fall below the cost of adjusting the institution. At that point optimization stops being a technical question and becomes a political one: whose discomfort gets treated, and whose conditions get changed?
The argument is not that treatment is a plot
Before going further, it is worth saying plainly what the case for alarm is not. Much of the distress people carry is genuinely disproportionate to its trigger, and much of it responds well to treatment. Depression, panic, trauma, and psychosis are not political inventions. The existence of a chemical-imbalance hypothesis that was oversold does not mean that psychiatric medication never helps; the literature on medicalization is a critique of overreach, not a denial of illness.
So the claim is narrower, and therefore harder to dismiss. There is a genuine category distinction between distress that is a symptom and distress that is an accurate appraisal, and a system that cannot tell the difference will apply the first label to the second when doing so is easier for whoever pays for the system. Lisa Cosgrove and colleagues made a related point in 2023 in Frontiers in Psychiatry, warning that the medicalization of distress, combined with a therapeutic culture that “responsibilizes” individuals for their suffering, shifts attention away from the upstream social causes of it. Their concern was pharmaceutical and diagnostic, not algorithmic, but the structural logic transfers cleanly to automated systems: a tool that treats the individual is a tool that can be pointed anywhere, including at the individual.
The philosopher Amia Srinivasan gives the moral spine of the argument in “The Aptness of Anger” (2018). Her target is the widely repeated claim that even justified anger is counterproductive and therefore should be set aside. Srinivasan argues that the counterproductivity critique often smuggles in a demand that the wronged party manage their own response for the comfort of others, and that this can constitute a form of “affective injustice.” Anger, on her account, is sometimes a correct appraisal of a wrong — it can be apt even when it is inconvenient. She does not claim that anger is always useful; she claims that its usefulness is a separate question from its aptness. A system optimized to reduce unpleasant affect will collapse those two questions into one, and it will collapse them in the direction of tranquility.
Where the technology changes the arithmetic
None of this requires artificial intelligence. Managers have reframed grievances as attitude problems, and doctors have prescribed sedatives for oppressive working conditions, for as long as either institution has existed. The reason machine learning matters is that it lowers the marginal cost of the individualizing move, raises its precision, and hides the choice inside a score.
Three distinct levels of capability are worth keeping apart. That a model can classify sentiment in text is demonstrated science and engineering, replicated many times over. That a model can infer a specific person’s emotional state from voice, face, or typing patterns well enough to act on it is plausible engineering, with real but contested accuracy, especially across cultures and individuals. That a system could reliably and safely decide whether a person’s distress is proportionate and then choose what to do about it is speculation, and it is speculation that quietly assumes away the very judgment problem described above. Most public discussion compresses all three into “AI detects emotions,” which is exactly the compression that makes the moral question invisible.
The workplace is where the engineering is furthest along and the stakes are easiest to see. In a 2024 study in Socius, Paul Glavin and colleagues surveyed a national sample of 3,508 Canadian workers and found that the perception of being surveilled was associated with greater psychological distress and lower job satisfaction, and that this ran through what they call stress proliferation: surveillance fed job pressures, reduced autonomy, and privacy violations, which in turn produced distress. In other words, the monitoring itself was a cause of the condition that monitoring is often marketed to detect. A dashboard that flags the exhausted worker is native to a system that may have made her exhausted.
Emotion AI sharpens this. In a 2023 CHI paper, Kat Roemmich and Florian Schaub studied how workers respond to systems that claim to read emotion, and found that the technology pushes people into additional emotional labor: workers reported managing and disguising their feelings precisely to protect what the authors call emotional privacy. A companion study by Corvite, Roemmich, and Rosenberg documented workers’ concerns about the same systems. The watershed here is subtle and important. A tool that claims to measure how you feel does not merely observe — it changes what you are willing to show. When that loop is closed, the measured state is no longer the person’s state; it is the person’s compliance strategy. And a system trained on the modified behavior will be confident about a variable it has itself distorted.
Distinguishing disorder from disproportion
The hard problem, and the one automated systems are least equipped to solve, is telling a symptom from an accurate reading. Roger Mulder’s 2014 editorial in the New Zealand Medical Journal, “Unmet need or medicalising distress?”, is useful because it works through a concrete case. A well-designed survey found that large fractions of university students screened positive for anxiety, depression, sleep problems, and harmful drinking. Mulder notes the awkward finding that fewer than two percent of those same students reported being dissatisfied with life, and that over eighty percent were satisfied. If nearly half the student body has a diagnosable disorder and nearly all of it is content, one of the measurements is doing something other than measuring illness. His worry is specifically about helplessness: pathologizing ordinary distress, he writes, tends to push people toward seeing themselves as reliant on professionals, and illness models “tend to attempt to relieve distress by focusing on individualised and private solutions rather than sociological or political explanations.”
This is the crux. The information required to distinguish a symptom from an accurate appraisal is often not in the person at all — it is in the situation. A model that sees only the individual, however much data it has about that individual, is structurally blind to the hypothesis that the workplace is genuinely intolerable. It can be extremely well-calibrated about your mood and still be unable to represent the cause. And the more fluent it becomes at describing your state, the more persuasive its recommendation to fix that state will sound.
The stakes of this failure are not only theoretical. When distress is reframed as a defect of the responder, the responder loses standing to object. The classic example is the earworm of the modern clinic: the physician who says the system is unsafe and is told to practice better self-care. Dean and Talbot’s own account is that clinician satisfaction became a “wellness” line item while the double binds that produced the injury were left in place. A remote monitoring system that detects nurse burnout and offers a breathing exercise is a technical descendant of that move.
Consent is real but weaker than it looks
The natural defense of these systems is consent. People opt in, the argument goes, and a voluntary tool cannot be an anesthetic. This defense is worth taking seriously, and then examining in context, because formal agreement is a thin protection under three conditions.
First, economic coercion. An employee who faces precarity is not choosing freely between using the system and not using it; the alternative to participation may be unemployment, and a preference formed under that pressure is not the same as one formed without it. Second, incapacity over the relevant horizon. A child cannot weigh long-term consequences of an intervention, which is why medical ethics treats surrogate consent as a special arrangement rather than an ordinary one. Third, and most corrosive, preference adaptation. If an intervention changes the preference that is later cited to authorize more intervention, then consent becomes circular: the system cites the docility it produced as evidence that it was wanted. Each of these is a familiar problem in clinical ethics, and none of them is solved by a check box.
Against this there is a real counterargument that deserves its weight. Refusing people treatment because their suffering is “really” structural is its own cruelty. A person in acute pain who cannot change the system that hurt them still needs help, and withholding an effective intervention in the name of political purity is exactly the kind of move that makes critics of medicalization look unserious. Srinivasan’s own point cuts this way: the aptness of an emotion does not settle what should be done with it. Treating a symptom honestly, to make a person capable of fighting the cause, is not an anesthetic. It is ammunition. The failure mode is not treatment. It is treatment that extinguishes the signal and calls the extinguishing a cure.
The question of who decides
The reason the anesthetic worry is a question about the state, rather than only about employers, is scale and legitimacy. Private tools can already do much of this. What state deployment adds is the authority to define the baseline — to decide what counts as a healthy emotional response, whose report is credible, and when an intervention may be imposed rather than offered. Those are questions of political philosophy, not merely design.
And they connect to a long-running asymmetry in how societies respond to dissent that is expressed as feeling. The comment that a policy is cruel can be reframed as anxiety; the demand that a wrong be corrected can be reframed as dysregulation. Once a system is authorized to score emotional stability, it acquires a quiet veto over the standing of a complaint, because a person marked as unstable can be discounted before their argument is even heard. This is a more serious harm than a bad mood, and it is one that a technical framing helps to conceal.
Designing for friction
It is possible to build these systems so they resist the anesthetic drift, though doing so is a choice against the grain of the economics. Several principles follow from the analysis above.
Ask what the state is responding to before asking how to reduce it. A system that cannot represent the situation is not entitled to conclude that the person is the problem. This is the difference between a tool that asks “what is this distress tracking?” and one that asks only “how do we lower this score?”
Expose the judgment. When a value is encoded in a threshold — “this level of anger is excessive” — that threshold is a policy, and policies should be visible and contestable rather than buried in a model weight.
Preserve the ability to say no, and to recover. Can a person disconnect, inspect the record, choose another system, retrieve their data, and return to baseline without punishment? If not, the tool may be technically voluntary while functionally sovereign.
Keep a named human responsible. Automation bias is real: as a system’s recommendations improve, deference becomes rational, and the competence to notice when it has failed erodes. A named person or institution accountable for consequential decisions preserves an avenue for that failure to be caught.
What would count against this worry
A reader should be able to say what evidence would weaken the case. If emotional inference systems turn out to be unreliable enough that they cannot be acted on at scale, the near-term risk shrinks, though the surveillance apparatus remains. If organizations that adopt these tools systematically reduce the upstream causes of distress — staffing, workload, autonomy — rather than shifting attention to the individual, the anesthetic mechanism is not in play. And if consent in practice turns out to be genuinely free and revisable, the coercion argument loses force. The strongest disconfirming evidence would be institutions using individual-level monitoring to detect problems and then fixing the problems. That would mean the technology was being pointed outward, at conditions, instead of inward, at people.
What the technology cannot settle
There is a temptation to treat moral questions as tractable once enough data is available. The desire is understandable and mistaken in this domain. Neuroscience can describe the correlates of a strong feeling without telling you whether the feeling is justified. The physical signature of anger looks similar whether the anger is apt or misplaced. The appraisal is a judgment about a situation, and judgment about situations is not the kind of thing a state classifier can settle, however accurate it is about the body.
The same limit applies to the state. A government may legitimately want a calm, functioning population, and may legitimately provide mental health care. What it cannot do without a moral cost is convert a well-founded objection into a treatable symptom, and then point to the treatment as the answer. The measure of a humane system is not how content it makes people under bad conditions. It is whether it leaves them able to tell the difference, and leaves them free to act on it.
Sources and further reading
- Amia Srinivasan, “The Aptness of Anger,” Journal of Political Philosophy 26(2), 2018 — https://onlinelibrary.wiley.com/doi/10.1111/jopp.12130
- Wendy Dean, Simon Talbot, and Austin Dean, “Reframing Clinician Distress: Moral Injury Not Burnout,” Federal Practitioner 36(9), 2019 — https://pmc.ncbi.nlm.nih.gov/articles/PMC6752815/
- Roger Mulder, “Unmet need or medicalising distress?,” New Zealand Medical Journal 127(1399), 2014 — https://nzmj.org.nz/journal/vol-127-no-1399/unmet-need-or-medicalising-distress
- Lisa Cosgrove et al., “Why psychiatry needs an honest dose of gentle medicine,” Frontiers in Psychiatry, 2023 — https://www.frontiersin.org/journals/psychiatry/articles/10.3389/fpsyt.2023.1167910/full
- Paul Glavin, Alex Bierman, and Scott Schieman, “Private Eyes, They See Your Every Move: Workplace Surveillance and Worker Well-Being,” Socius 11(4), 2024 — https://pmc.ncbi.nlm.nih.gov/articles/PMC11300163/
- Kat Roemmich and Florian Schaub, “Emotion AI at Work: Implications for Workplace Surveillance, Emotional Labor, and Emotional Privacy,” CHI 2023 — https://dl.acm.org/doi/10.1145/3544548.3580950
- K. S. Jacob, “Medicalizing Distress, Ignoring Public Health Strategies,” Indian Journal of Psychological Medicine 36(4), 2014 — https://pmc.ncbi.nlm.nih.gov/articles/PMC4201784/
Loading comments…