On August 1, 2012, a software deployment problem at Knight Capital helped turn a trading system into a source of enormous losses. The SEC’s later order described a defective deployment, inadequate controls, and millions of executions during roughly forty-five minutes. The firm realized a $460 million loss. This was an automated trading failure, not a generative-AI incident. It remains a concrete example of what happens when speed and authority outrun the controls around them. SEC order, 2013.
AI leverage has the same general asymmetry: the mechanism that multiplies useful work can multiply an error. A wrong classification can be copied into many records. An inaccurate promise can reach many customers. A mistaken purchasing rule can commit more cash before a person notices the first result.
The relevant question is not whether a model is occasionally wrong. Every operating system contains errors, including human ones. The question is how far an error can travel, how much authority it encounters, how quickly it becomes visible, and what the business can still recover when it does.
The path from a small mistake to a large effect
An error becomes consequential through a sequence. A source is wrong or misunderstood. A system accepts it. A decision uses it. An action changes something. Other actions depend on that change. The affected person or system responds. Recovery may then require more than reversing the original field.
Consider a hypothetical retailer whose assistant incorrectly labels an item as compatible with a popular machine. The error first appears in a draft listing. If a reviewer catches it, the loss may be a few minutes. If the listing is published, buyers may rely on it. If another assistant uses the listing to answer questions, the claim spreads. If the purchasing system sees increased interest and orders more stock, the error begins consuming capital.
The initial classification has not become more wrong. Its consequences have become larger because it crossed more boundaries. This is why a system’s average answer quality does not fully describe its risk. Authority, repetition, dependencies, and timing determine the loss pathway.
A useful design examines that pathway before granting more autonomy. Where can the claim be checked? Which step creates a commitment? Which downstream system treats the output as authoritative? What evidence identifies affected cases?
Repetition changes the scale
A person making one manual error may affect one task. An automated process can repeat the error across a batch before the operator sees a result. The process can also repeat a correct action unintentionally when a retry is mistaken for a new request.
A hypothetical system might send the same discount offer twice because it cannot tell whether the first send succeeded. A stock update could subtract the same sale more than once. A purchasing assistant could create duplicate drafts that a busy reviewer approves separately.
The protection is to preserve task identity and state where the connected systems support it. Distinguish a new action from a retry. Record whether an action is prepared, approved, attempted, accepted, or reconciled. When success is unknown, investigate before repeating a consequential action blindly.
Provenance and auditability explains the evidence trail. Here the purpose is loss containment: the business needs to identify the affected actions quickly enough to prevent the first mistake from becoming a batch of obligations.
Connected agents can share the same wrong premise
Several agents may look like several independent checks. They can also act as several copies of one mistaken assumption. If each uses the same inaccurate source, prompt, or rule, agreement may increase confidence without adding independent evidence.
Imagine a hypothetical product workflow. One assistant drafts a claim, another approves the wording, and a third prepares a campaign. If all three use the same unverified compatibility field, their agreement does not establish compatibility. The workflow has checked presentation, not the underlying fact.
Independence should be defined by the failure being tested. A deterministic check against a verified specification can be more useful than another general model opinion. A human inspection may be necessary when the source record does not capture the physical condition.
Multiple agents can still be valuable for different jobs. The important boundary is that the system should not count repeated reliance on the same evidence as new evidence. Name the authoritative source and identify what each check actually establishes.
Authority determines the possible damage
A model allowed to draft a recommendation can cause confusion or consume review time. A model allowed to change a public page can influence customers. A model allowed to spend money, delete records, or make binding commitments can create larger and less reversible consequences.
This is why permission design belongs in the economic case. The business should grant only the authority needed to produce the expected value. Additional convenience has to justify the additional exposure.
OWASP’s versioned excessive-agency guidance distinguishes excessive functionality, permissions, and autonomy. Those are different routes by which unexpected or manipulated model output can become damaging action. The guidance supports scoped tools and enforcement in connected systems; it does not promise a complete defense against every attack or mistake. OWASP LLM06:2025.
A hypothetical inventory assistant may need to read stock and propose a reorder. It may not need access to bank transfers or the ability to delete customer records. Keeping those capabilities unavailable reduces the loss pathways regardless of how fluent or cooperative the assistant appears.
The same error can move through different channels
Errors can affect money, information, relationships, physical work, or reputation. A monetary spending cap protects one channel but may leave another open.
An assistant could remain below its spending limit while emailing private information to the wrong recipient. It could preserve private data while making a false public claim. It could avoid external communication while changing a record that later drives an unsafe physical decision.
Map the consequential outputs rather than assume one budget controls everything. Which records can change? Who can receive a message? What commitments can be made? Which physical actions depend on the result? Which public statements carry the business’s name?
For each output, identify a limit and a check appropriate to the consequence. A draft-only boundary, approved recipient list, scoped record access, or human review may be more useful than a general instruction to be cautious. Agent permission architecture develops those mechanisms in their proper technical home.
Fast feedback can arrive too late
A dashboard may update every minute and still fail to protect the business. If consequential actions happen faster than the signal reveals their outcome, the process can accumulate exposure before the operator sees a warning.
A hypothetical campaign may produce immediate clicks while refunds and complaints arrive weeks later. A purchasing process may show orders placed today while inventory becomes unsellable months later. A customer-service assistant may appear fast while inaccurate promises create future disputes.
Define the delay between action and informative feedback. Limit expansion during that delay. A pilot can begin with a small batch, inspect accepted results, and wait for the relevant consequences before increasing volume.
This is not an argument for waiting forever. Some decisions have useful early indicators. The task is to distinguish an early indicator from the final outcome and make the exposure appropriate to what is actually known. A fast signal of activity cannot substitute for a slower signal of value.
Reversibility has a boundary
A changed draft can usually be restored. A public message can be corrected but not unread. A payment may be difficult to recover. A disclosed secret cannot be made secret again. A physical action may leave damage even after the system stops.
The business should classify actions by practical reversibility before delegating them. Ask what restoration would require, who would be affected, and whether the original state remains available.
A rollback plan is useful for software or data changes, but it does not erase external obligations. If an assistant sent a mistaken price to a customer, restoring the website may not resolve the customer’s reliance. If it approved a shipment, reverting an order record does not bring the package back.
The kill engine separates stopping new work, rolling back a version, reducing scope, and retiring a process. This article adds the cascade question: which consequences continue after the first step stops, and how will the owner identify and address them?
Contain the first uncertain action
A staged release can reduce the area affected by an error. Begin with drafts or copies. Move to a limited set of ordinary cases. Review difficult cases separately. Increase scope only when the evidence supports the next boundary.
For the hypothetical retailer, a new compatibility workflow might first generate draft recommendations for a small product family. A reviewer checks them against verified specifications. The business then examines published pages and resulting questions before allowing broader use.
The useful limit is tied to consequence, not merely the number of tasks. Ten low-cost drafts and ten high-value commitments create different exposure. A small sample can still be dangerous if it contains an irreversible action.
Keep a way to stop new work independently of the model. Preserve the accepted prior version and the records needed to identify changes. Decide who can invoke the stop and who can authorize resumption. A control nobody knows how to operate is a diagram, not protection.
A small business needs a proportionate recovery plan
A recovery plan does not need a large incident-response department. It needs enough clarity for the operator to act under pressure.
Name the first action: stop the relevant workflow or revoke its authority. Identify the evidence to preserve before cleaning up. Locate affected records and external commitments. Decide who must be informed and which remedies or corrections are appropriate. Restore a known process and verify it before resuming.
Avoid erasing the trail in a rush to make the system look normal. The business may need the original record to understand the failure, identify affected people, or support a remedy. Corrections should be visible rather than silently rewriting history.
After the immediate work, separate causes. Was the source wrong? Did a rule misinterpret it? Was the permission too broad? Did monitoring fail? Did the operator ignore a signal? A single label such as “hallucination” can conceal several different problems and lead to the wrong repair.
Human oversight must be able to change the outcome
A person technically included in the workflow may have little practical control. They receive too many approvals, cannot inspect the evidence, lack time to review, or are rewarded for accepting recommendations quickly.
Effective oversight provides the relevant record, a clear decision, enough time, and authority to reject or stop. The reviewer should understand which mistakes matter and where the system is least reliable. A checkbox at the end of an opaque process does not supply that capability.
A hypothetical purchasing manager reviewing dozens of nearly identical drafts may stop noticing an unusual quantity. The workflow can help by highlighting deviations from approved limits and grouping ordinary work separately from exceptions. The business should not require the person to discover every important difference through sheer vigilance.
The strongest design combines usable human judgment with independent boundaries. A person can evaluate the unusual case while the system prevents actions beyond a hard limit. Neither layer should be treated as perfect, and the recovery plan should account for failures of both.
Count benefits and harms on the same system boundary
A project may save time in drafting and create more work in support. It may reduce one class of error while increasing another. A fair evaluation follows both effects through the same operating process.
For a hypothetical assistant answering product questions, record accepted resolutions, review time, escalations, wrong promises, and later complaints relevant to those answers. Do not count the quick replies as benefit while assigning the resulting corrections to somebody else’s cost center.
Hallucinated profitability addresses the numerical claim. The risk question is broader: does the added capability remain worthwhile when failures, monitoring, and recovery are included? Some projects will. Others will need narrower authority or a different process.
A single dramatic case should not establish that all automation is dangerous, just as one smooth demonstration should not establish that all autonomy is safe. Use cases to identify mechanisms, then test the mechanisms in the task and conditions the business actually faces.
Practice the recovery without creating the harm
A tabletop exercise can reveal gaps before the business experiences a failure. Use a hypothetical event or copied records and walk through the response.
Suppose the inventory assistant changed compatibility on twenty listings before an error was noticed. Who can stop further changes? Which version contains the previous values? Which customers received the claim? Which orders depend on it? Who decides on corrections and remedies? What must be verified before resumption?
The exercise should identify concrete missing capabilities. If the operator cannot locate affected changes, improve the record. If stopping one workflow leaves another using the same bad field, expand the containment plan. If the remedy requires a person who is unavailable, assign a backup.
Label the exercise accurately. It is preparation, not evidence that the business has survived the corresponding real-world event. Record what the exercise established and what would still depend on production conditions.
AI Leverage in Practice
What changed: one error can now move through more digital work with less friction. Useful capability and possible loss both depend on the authority and connections around the model.
What to do today: choose one consequential workflow and draw its error pathway from source to decision to action to downstream reliance. Mark the first external commitment and the first practically irreversible effect.
Put a limit at the earliest useful boundary. Use scoped authority, a limited initial batch, preserved versions, and a stop control someone can operate. Run a recovery exercise on copies and identify affected cases without relying on the assistant’s narrative alone.
Evaluate the saving alongside review, failure, and recovery work. Keep the result narrow enough to match the evidence. Increase exposure only when the next stage remains within the business’s capacity to inspect and recover.
What may come later: improving models may reduce particular errors. More connected agents may create new cascade pathways. Better capability is a reason to reconsider the design, not remove every boundary around action.
Leverage the capacity to recover
The Knight Capital case is a historical warning about software speed and control, not a prediction that every small AI workflow will suffer an equivalent event. Its usefulness lies in the mechanism: a failure becomes much larger when rapid action meets insufficient boundaries.
A responsible AI system gives the business more ability to do useful work and more ability to understand, stop, and correct its work. The owner should know where an error can travel and which commitments remain after the system is paused.
Leverage is strongest when the business can survive being wrong, learn why, and continue serving people responsibly.
Find the complete Age of AI Leverage series and wider guidance in the AI section.
Sources
- SEC, In the Matter of Knight Capital Americas LLC, October 16, 2013. Historical non-AI software-control failure; the order’s findings support the dated account.
- OWASP, LLM06:2025 Excessive Agency.
- NIST, AI Risk Management Framework, voluntary risk-management guidance; no certification claimed.
Loading comments…