> For the complete documentation index, see [llms.txt](https://pharos-production.gitbook.io/pharos-production-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://pharos-production.gitbook.io/pharos-production-docs/recover-ai-tool-call-without-repeating-effect.md). # Recover an AI Tool Call Without Repeating Its Effect A proposed recovery runbook for lost tool replies: retain operation identity, inspect provider evidence and authorize any further external action.
Proposed AI tool recovery sequence: hold automatic dispatch, recover the same operation identity and verify the provider result. Fictional R-73; no external tool call or recovery test was run.

Proposed R-73 runbook. Fictional example; no external tool call or recovery test was run.

An AI agent sends a request that changes an external system. The response does not arrive before the client times out. The agent still needs a result, but another call may repeat an effect that already happened. Recovering the connection and recovering the operation are different jobs. Start by suspending automatic dispatch for that operation. Recover its identity, inspect the provider's retry contract and look for an authoritative result. Only then choose whether to return a known result, continue a permitted attempt or escalate an unresolved outcome. A new request identifier is not a recovery strategy for an unknown effect. This guide provides proposed runbook R-73 and a fictional work-item update. It demonstrates an operating sequence and review questions, not a tested integration. No external tool call, provider transaction, database write or client recovery exercise was executed for this article. ## Establish what the missing response actually tells you A local timeout establishes that the caller did not obtain the expected response within its waiting period. It does not establish where the request stopped. The request may have failed before dispatch, reached the provider without executing or completed while its response was lost. Preserve those possibilities until evidence narrows them. Use separate fields for the local observation and remote outcome. For example, the local observation can be RESPONSE\_NOT\_RECEIVED while the remote outcome remains UNKNOWN. Do not replace UNKNOWN with FAILED merely because the application's exception handler received a timeout. That translation would turn missing evidence into permission to repeat a potentially completed action. Also separate the provider operation from the business task. An external update may have completed while the agent failed to save its response locally. The application may still need to report completion, update its own record or resume a later step. Those remaining tasks do not necessarily require another external mutation. The proposed runbook assumes the team is authorized to inspect the relevant operation and make the recovery decision. It does not expand an agent's permissions. Where the action changes customer communications, access or another sensitive business state, use the authority already required for that action and its recovery. ## Step one: suspend automatic attempts for the same operation Prevent another worker or agent continuation from dispatching the operation while the recovery coordinator investigates. Record the hold against the operation identity, not just the current process. Restarting an agent should not erase the fact that an earlier attempt may still have an unresolved external effect. A hold needs an owner and a review condition. State who will inspect the evidence, when the review is due and what event permits another action. A permanent unexplained hold creates abandoned work. An automatically expiring hold can create duplicate work if expiry silently authorizes a fresh call. Use a coordination mechanism appropriate to the system, such as a durable operation state and an exclusive dispatch claim. The runbook does not prescribe a database product or claim that a lease alone prevents external duplication. A previous worker may still be executing after another process decides its lease has expired. Preserve the original request evidence before changing any state. Capture the dispatch time, intended effect, available provider references and the last local observation. Keep authentication secrets and unrelated customer data outside the recovery record. The coordinator needs a useful evidence path, not a copy of every sensitive field in a request. ## Step two: recover the identity before reconstructing the request Find the stable operation identifier created before the original external call. Keep it distinct from the attempt identifier. One intended operation can have several transport attempts under a provider-supported retry contract. Creating another attempt should not silently create another business operation. Record the original idempotency key where the provider supports one, plus the endpoint, relevant account scope and request parameters. A locally derived payload fingerprint can help detect a changed request. It does not prove that the provider uses the same comparison rule or that a different serialization has equivalent business meaning. If the operation identity was never retained, treat that as a recovery limitation. Do not manufacture a replacement key and describe it as the original. Search for provider evidence using authorized references that actually exist, then escalate if the relationship to the intended operation cannot be established. The identity should survive the failure being investigated. A key stored only in an agent's transient conversation may be unavailable after a restart. A useful implementation acceptance question is whether a receiving worker can recover the same operation, request scope and previous observations without asking the original process to remember them. ## Step three: read the provider's contract for this operation Inspect the actual API and version used by the tool adapter. Identify whether idempotency applies, how the provider matches requests and how long its protection remains available. An adapter's generic retry setting cannot replace those details. Some operations may have no provider-supported deduplication mechanism. Stripe's [idempotent requests documentation](https://docs.stripe.com/api/idempotent_requests) states, specifically for API v1: > we save results only after the execution of an endpoint begins. That boundary matters. The same documentation distinguishes validation and concurrent-request conflicts before execution from results saved after execution starts. It also describes parameter matching and key retention. These are examples of a particular provider contract, not guarantees that can be transferred unchanged to another tool or API version. For an actual adapter, retain a dated reference to the applicable rules. Check whether a retry returns a saved response, resumes partial work or does something else. A saved error response may still leave a business outcome requiring investigation. Receiving a response on the second transport attempt does not automatically establish that the intended business task succeeded. Review the retention window before any retry. [Stripe's low-level error guidance](https://docs.stripe.com/error-low-level) explains that reusing a pruned key can execute a new request and instructs callers to reconcile unresolved outcomes rather than keep retrying outside the stated window. Other providers can have different rules. The runbook must bind the decision to the contract actually in force. ## Confirm which recovery capabilities the adapter actually exposes Before assigning the runbook to an operating team, check that its proposed evidence paths exist. The adapter may send an idempotency key while exposing no result lookup to the coordinator. It may retain a provider identifier that only the supplier can inspect. These are different operational capabilities and should be reviewed separately. Ask the implementation owner to demonstrate how an authorized reviewer retrieves the original request scope, follows a provider reference and identifies a terminal result. This is a request for future evidence, not a claim that those capabilities have been demonstrated here. Record unavailable access as an open dependency with a receiving owner. Keep the provider contract and adapter behavior distinct. A provider may support safe retries while the adapter generates a new key every time a worker restarts. Conversely, a stable local identifier cannot create provider deduplication where the endpoint has none. The runbook needs both boundaries to be understood before another attempt is permitted. State what the agent can do without escalation. It might be allowed to retrieve a status and return a known result while a mutating retry remains reserved for an authorized operator. That division can be useful even when fully automatic recovery is inappropriate. A missing automation capability should not be concealed by broadening the agent's authority during an incident. ## Step four: check the authoritative result before selecting an action Use the provider's supported result lookup, object state or other documented evidence path. Record what was checked and when. A local log saying that a request was sent proves less than a provider record identifying its completed effect. Conversely, an unavailable lookup is not evidence that no effect occurred. Interpret a negative lookup under its documented scope. Search filters, account boundaries, propagation delay and missing permissions can affect what the reviewer sees. If the lookup cannot establish absence for this operation, retain UNKNOWN. Do not convert a convenient empty search result into a universal guarantee that another call is safe. Tie the result to the original request. A provider object with a similar name may belong to another operation. Use the identifiers and scope available under the actual contract. Where a status must be reviewed manually, retain the evidence reference and the person who accepted its interpretation rather than silently treating the search as an automated proof. Write down the outcome in terms that another operator can inspect. Known completion should include the relevant result reference and acceptance scope. Known non-execution needs evidence supporting that conclusion. An unresolved outcome needs the remaining question, its owner and the action currently withheld. ## Step five: choose a bounded recovery action The following decision table is a proposed operating aid. It does not replace provider documentation or the application's authorization policy. Every action remains conditional on the specific request, evidence and contract. An unavailable prerequisite keeps the recovery decision open. | Observed condition | Proposed next action | Required boundary | | -------------------------------------------------------------- | ------------------------------------------- | ------------------------------------------------------------------- | | Original effect is authoritatively confirmed | Recover the known result locally | Confirm its identity and accepted scope | | Request demonstrably never left the local system | Consider dispatching the original operation | Verify no competing attempt was dispatched | | Result is unknown and a supported retry contract remains valid | Consider a bounded same-operation retry | Same request scope, permitted key use and approved retry budget | | Result is unknown and key protection has expired or is unclear | Reconcile or escalate | Do not create a new operation to clear the timeout | | Request parameters have changed | Open a new decision | Establish how the changed intent relates to the unresolved original | | Provider result and local record disagree | Investigate the discrepancy | Preserve both records; do not overwrite the inconvenient one | | A compensating action is proposed | Review it as a separate effect | Obtain the authority and evidence required for that action | A permitted retry should have a specific limit and a receiving owner if it fails again. Do not let a natural-language agent invent a wider retry budget because the task remains unfinished. Keep transport resilience inside the adapter's approved contract and business recovery inside the operating decision. Record the selected action before executing it. Include the evidence used, the applicable contract reference and the authorized actor. That record lets the next worker distinguish a planned action from a completed one. If the process stops again, another reviewer should be able to find the last accepted boundary. ## Walk through fictional R-73 without assuming a second effect Suppose an internal agent is authorized to apply a label to work item W-73. The tool adapter records operation OP-73, attempt A-1 and a request fingerprint before dispatch. The client then records RESPONSE\_NOT\_RECEIVED. The provider outcome remains UNKNOWN. These identifiers and observations are fictional; no work item was updated. The coordinator places OP-73 on hold and locates its original request scope. A proposed provider lookup would search for evidence tied to that operation, not merely check whether W-73 currently has the label. Another actor might have added the same label for a different reason. The current field value alone may not establish the original request's outcome. Assume the fictional review obtains a provider reference showing that OP-73's intended update completed. The runbook then directs the team to recover that known result into the local operation record. It does not direct the agent to submit another label update. Any later business step needs its own prerequisites and authority. If that provider evidence were unavailable, the decision would be different. The coordinator would inspect the applicable retry contract and its remaining window. Without sufficient evidence or a valid permitted route, OP-73 would stay unresolved and move to the assigned reviewer. The example deliberately does not claim that every lost response can be resolved automatically. | Fictional R-73 record | Before review | After the assumed evidence review | | ------------------------------ | ---------------------------- | -------------------------------------------------------------- | | Operation identity | OP-73 retained | OP-73 unchanged | | Attempt | A-1 has no received response | A-1 linked to the recovered result reference | | Remote outcome | UNKNOWN | Completion accepted only under the fictional provider evidence | | Additional external mutation | Withheld | Still withheld; recover the known result locally | | Real execution in this article | None | None |
Fictional R-73 decision route. OP-73 is retained while A-1 has no reply. A confirmed known result is recovered locally without another mutation. An unresolved outcome requires checking the retry contract and authority, or reconciliation and escalation. No operation was executed.

Fictional R-73 evidence route. The example assumes a result review; it reports no executed operation.

## Coordinate concurrent workers without confusing leases with outcomes Two workers can observe the same unresolved operation. The implementation needs a rule for which worker may dispatch an attempt or record an accepted recovery decision. Keep that coordination state durable enough for the intended failure model. A note in one worker's memory is not a shared operating boundary. Treat a dispatch claim's expiration separately from the provider outcome. Expiration may allow another worker to investigate, but the first worker could still have sent a request. The next worker must recover the operation history and applicable provider evidence before deciding whether it can dispatch another attempt. Local atomic updates can protect local state transitions. They do not create an atomic transaction with an unrelated external service unless the integration actually supplies that contract. Record the boundary between a local commit and the external effect. This is where a crash can leave local and remote observations temporarily inconsistent. Define how contradictory updates are handled. A late response from A-1 may arrive after a recovery coordinator starts reviewing the same operation. Preserve it as evidence associated with A-1. Do not silently let whichever process writes last decide the business outcome without applying the operation's state and review rules. ## Treat changed intent and compensation as separate decisions A user may change the requested action while the original result is unknown. For example, a different label or target item changes the request scope. Do not reuse the old key with altered parameters merely to keep the conversation moving. Establish the relationship between the changed intent and the still-unresolved operation first. A compensating action also produces an effect. Removing a label, revoking access or reversing another update can be inappropriate if the original operation never happened or another actor has since changed the record. Review the current state and the authority required for the compensation. Do not describe compensation as a universal rollback of history. Keep the original evidence when opening the new decision. Link the new intent or compensation request to the earlier operation without rewriting its result. The record should distinguish what the system knows about the original effect from what someone now wants it to do next. The agent's explanation should preserve that distinction for the user. It can report that the original outcome remains unresolved and that another action requires review. It should not announce success because a recovery ticket was created or failure because a timeout exception was recorded. ## Review the runbook with proposed acceptance scenarios Before using R-73 in production, agree which observations a controlled test must produce. The table below proposes scenarios and expected review questions. None of these tests was executed for this article. A completed worksheet would need actual results, configuration references and an authorized reviewer. | Proposed scenario | Evidence to inspect | Review question | | ---------------------------------- | ----------------------------------------------- | ------------------------------------------------------------------------- | | Crash before dispatch | Durable operation and dispatch records | Can the team establish whether any attempt left the system? | | Effect completes but reply is lost | Provider result and local missing response | Can the known result be recovered without another effect? | | Two workers investigate | Dispatch claims and operation history | Can only an authorized attempt proceed? | | Key window expires unresolved | Original time, provider contract and escalation | Is blind replay withheld? | | Parameters change during recovery | Original and proposed request scopes | Does the change open a new decision? | | Result arrives late | Attempt identity and recovery record | Is the observation retained without silently replacing accepted evidence? | Include limits in each test record. A local fixture can validate state transitions without demonstrating a real provider's idempotency behavior. A sandbox test can support the configuration and scenarios actually exercised there. Neither should be reported as a universal guarantee that duplicate business effects are impossible. Make the acceptance owner explicit. The adapter developer can explain the implementation, while the operating team must understand the evidence and permitted actions it will receive. If the receiving team cannot retrieve a result or contact the provider, the runbook still has an unresolved operating dependency. ## Keep the recovery clock separate from the request clock Record the original dispatch time, the local waiting deadline and the time each later observation was checked. These timestamps answer different questions. A timeout recorded locally does not establish when the provider began work, and a later lookup does not change the original dispatch time. Where the provider's retry protection has a documented time boundary, identify the rule used to calculate the decision window. Preserve the source and any unavailable clock information. Do not extend the window because an agent conversation restarted or because a recovery ticket was assigned to another team. Give an unresolved case a review deadline without turning that deadline into automatic permission to repeat the effect. The next review may obtain new evidence or escalate a provider question. It may also conclude that the outcome remains unknown. That is a meaningful operating result when it prevents an unsupported external action. ## Hand over unresolved work with its evidence path A recovery handover needs more than a ticket marked retry pending. Provide the stable operation identity, last attempted action, known observations and the next decision currently withheld. Identify the person or team accepting coordination. The receiving owner should be able to open the retained evidence under the agreed access controls. Keep the business outcome separate from administrative closure. A coordinator can finish a handover while the provider outcome remains unknown. A developer can repair a timeout-handling defect while a particular work item still requires reconciliation. Close each responsibility only under the evidence appropriate to that responsibility. Review the runbook after a contract or adapter change. A new endpoint version, different retention rule or altered request schema can invalidate a previously permitted retry path. Preserve the old record as history and reopen the relevant acceptance questions before relying on the new configuration. ## Scope implementation around the recovery boundary [Pharos Production AI agent development services](https://pharosproduction.com/services/ai-agent-development/) is a relevant starting point for discussing tool integration and the application behavior around it. Bring one redacted operation trace, the provider contract and the unresolved decision. Those inputs support a more specific implementation discussion than a broad request to make an agent reliable. Apply the same acceptance questions to a Pharos Production tool-recovery proposal as to any other implementation. Ask who retains operation identity, who may dispatch again and what evidence supports the outcome. A public service description does not establish that your provider integration has passed these checks or that recovery is included in an existing support agreement. A useful delivery scope can include the runbook, adapter changes and agreed evidence from controlled scenarios. Price and accept those deliverables under the actual engagement. Keep a proposed test plan separate from completed tests, and keep both separate from real production results. The next action after a lost tool response should follow the operation's evidence and authority. Preserve the original identity, establish the provider's rules and recover a known result where possible. When the effect remains uncertain, retain that uncertainty and assign its next decision before permitting another external action. ## More insights to read * [AI Agent Tool Permissions: A Side-Effect and Approval Envelope](https://pharosproduction.blogspot.com/2026/10/ai-agent-tool-permissions-side-effect.html) * [Application Support Acceptance: Requests, Releases and Recovery](https://pharosengineeringnotes.wordpress.com/2026/11/01/application-support-acceptance-requests-releases-and-recovery/) * [Queue Migration Acceptance: Residual Work and Final Ownership](https://pharosengineeringnotes.wordpress.com/2026/11/07/queue-migration-acceptance-residual-work-final-ownership/) * [Legacy Cutover Fieldbook: Snapshot, Reversible Step and Repair Path](https://hackmd.io/@dmytro-nasyrov/legacy-cutover-fieldbook-snapshot-reversible-step-repair-path) * [AI Agent Integration Estimates: Price the Connector Failure Paths](https://pharosproduction.blogspot.com/2026/10/ai-agent-integration-estimates-price.html) ## About the author
Portrait of Dmytro Nasyrov wearing a dark suit and light blue shirt against a dark background.

Dmytro Nasyrov. Photo supplied by the author.

*Written by* [*Dmytro Nasyrov*](https://pharosproduction.com/dmytro-nasyrov/) *PhD, software architect with 24 years of production experience. Dmytro is the founder and CTO of Pharos Production. He works on production software architecture for FinTech, AI, Web3 and blockchain systems.*