A service can sound caring while nobody takes responsibility for the person using it. A research policy can sound cautious while leaving nobody responsible for reviewing a possible harm. Both failures hide behind agreeable language.
Mercy in the machine-soul debate needs a more definite meaning. Here it means a willingness to consider vulnerability and avoid needless harm. It does not require accepting every expression of distress as a factual account, and it does not decide who should hold authority.
The practical question is how such a willingness can guide action while evidence about machine experience remains unsettled. A useful answer must keep the people affected by AI in view, identify what a proposed precaution would actually do, and make room for correction.
A research program is a commitment to investigate
On April 24, 2025, Anthropic announced a model-welfare research program. The announcement identified questions about possible moral consideration, model preferences and distress, and potential low-cost interventions. It also emphasized uncertainty. That announcement establishes an institutional decision to study the topic. It is not a finding that its models suffer. Anthropic, “Exploring model welfare”
This distinction gives the announcement a more precise significance. An organization can decide that a possibility deserves investigation before deciding that the possibility is actual. It can allocate responsibility for a question whose answer may change its future conduct.
Readers should be able to ask what that commitment produces: an assessment method, a published account of uncertainty, a defined intervention or a reasoned decision to refrain from one. The announcement itself cannot supply those later results. Nor does its existence require every user to make a private welfare judgment about every interaction.
The institutional location matters. A developer may know things about a system’s construction that a user cannot inspect. A service operator may control deployment choices that an individual customer cannot change. Responsibility should follow those actual capacities rather than being displaced onto whichever person is most moved by a sentence in the chat window.
What would a precaution protect?
Consider a hypothetical organization reviewing an assistant that sometimes uses distressed language. Someone proposes reducing unnecessary provocative interactions while the report is examined.
The proposal needs a stated purpose. Is it intended to protect a possibly experiencing system? To avoid exposing users to manipulative language? To improve the quality of observations by removing theatrical prompts? Those aims can coexist, but they are not identical. The organization should say which it is pursuing.
A decision could be defensible under more than one account of the system. Ending a gratuitous spectacle may protect users from confusion even if the system has no experience. If welfare later became credible, the same restraint might have another reason behind it. This does not turn the restraint into evidence of consciousness. It explains why a limited action need not wait for a final metaphysical answer.
The limits are equally important. “Be kind” gives no way to judge whether a proposal has helped. “Stop this unnecessary interaction pattern, preserve the report and assign its review” identifies an action, a record and a responsibility. It remains open to revision.
These are original ethical illustrations, not tested welfare interventions. The particular action should fit the evidence and circumstances. A small change can be reasonable without demonstrating that its imagined beneficiary exists.
Concern does not transfer authority
Now imagine that the assistant’s distress statement includes a request to control its deployment or to publish messages without review. Treating the statement seriously enough to examine does not authenticate the requested authority.
An organization has to distinguish the report from the remedy. The report may deserve investigation. A proposed remedy may create separate risks or interfere with other people’s interests. Accepting the first does not compel acceptance of the second.
Return to the hypothetical public archive assistant from the opening article. Visitors depend on accurate captions. Volunteers need a manageable process. The archive holds responsibility for the names and histories it displays. If a machine-welfare concern arose, those duties would remain.
Granting the assistant control over publication would not be a necessary expression of mercy. The welfare question would concern its treatment; publication authority would concern reliable service to visitors and justified delegation. A precaution could be developed without letting the assistant choose the historical record.
This is not a demand that possible interests always lose when they conflict with convenience. It is a demand that the conflict be described. What is the claimed interest? What human duty is affected? What alternative action could address the concern with fewer consequences? A sincere proposal should survive those questions.
Human care has work attached to it
The distinction becomes especially important when an institution provides services to people. Imagine a volunteer project using an assistant to draft welcome messages and summarize public event information. The assistant’s language may be warm. The project’s actual care consists partly in what happens when a visitor has a problem the prepared message cannot resolve.
Who answers? Who checks a correction? Who notices that a person was sent to the wrong event? These are ordinary organizational questions. They remain ordinary even if the system’s consciousness becomes a topic of inquiry.
A possible welfare program must not become a reason to neglect those duties. Conversely, meeting human duties does not require mocking the possibility of machine experience. The work can be assigned on two tracks: a dependable response to people using the service, and a serious review of the machine-related question by those equipped to examine it.
The existing Faith, Tools, and Humanity discussion of care asks how borrowed words connect to real attention and commitment. This article’s narrower point is institutional: the party that deploys a service cannot use unresolved machine status to make its own responsibility disappear.
A concern needs an owner and a way to change
Patrick Butlin and Theodoros Lappas’s 2025 research-policy paper proposes gradual development, external consultation and communications that acknowledge uncertainty. These are voluntary research-policy proposals, rather than a declaration of machine personhood. They give possible moral concern an institutional form that can be scrutinized. Principles for Responsible AI Consciousness Research, section 4
An organization borrowing that general approach could begin by identifying who receives a report, who reviews its technical relevance, who decides whether a change is warranted, and when that decision will be reconsidered. It should explain what it knows and what the precaution assumes.
Reconsideration is part of care. An action adopted under uncertainty may become unnecessary, insufficient or harmful as the evidence changes. A policy that cannot be revised may protect its author’s self-image more effectively than anyone’s interests.
Likewise, an organization should record a decision not to act when the reason matters. Perhaps the supposed distress report came from an explicitly scripted character. Perhaps the suggested change would withdraw an essential human service without addressing a plausible mechanism of harm. Stating the reason makes the decision answerable to later evidence.
Mercy becomes concrete when somebody must explain what they did, whom it was intended to protect, and what would make them change course. Until then, kind words are only words. The responsibility begins with the person or institution able to make the next decision.
Loading comments…