# Recover after reconnect

Rebuild a bot's orders, executions, balances, and triggers after a disconnect or restart without duplicating orders.

Pause new quoting while you reconnect and reconcile account state. Orders can fill or end while your bot is offline, triggers can place children, and a lost command reply can leave its outcome unknown. Resume once the affected state is current and your pending intents are resolved.

Use `https://api.testnet.polyester.com` for reads and `wss://api.testnet.polyester.com/v1/ws/private` for private order updates. The trade socket uses the host `trade.api.testnet.polyester.com`. Testnet is the public environment; Mainnet is coming soon.

## What can be lost

A reconnect can replay duplicate events or leave a gap. When the private order socket cannot resume, its fresh snapshot replaces order state but does not replay the intervening amend, cancel, or rejection outcomes. Fills remain available from trade history. The snapshot includes open, pending, and recently ended orders, not a complete archive.

Keep a durable journal of submitted intents, client order IDs, assigned order IDs, and replies. Save stream checkpoints atomically with the data they describe. A restart without the matching local state requires a fresh snapshot and history reconciliation, even if a cursor survived.

## Detect a gap

The [private WebSocket](https://testnet.polyester.com/docs/developer-docs/connectrpc/private-websocket) carries your orders, and the realtime connection in the [WebSocket session model](https://testnet.polyester.com/docs/developer-docs/shared-concepts/websocket-session-model) carries the event streams. Each signals lost state differently:

- **Order sequence**: if private order `seq` skips or repeats, drop that subscription and resubscribe with the last applied cursor.
- **Order resync**: on `resyncRequired` or a changed snapshot `epoch`, install the fresh snapshot and refetch history.
- **Realtime recovery**: on `recovered=false` or a stream-position gap, backfill the affected stream through its read API.
- **Order version**: when a read has a higher `version`, keep the newer state and reconcile any missing executions.

Order versions compare copies of one physical order: `0` means accepted but not yet on the book, then published changes increase the version. They are not account-wide sequence numbers. A jump can reveal missed intermediate states; equal versions do not prove you received every fill or command outcome. Stream positions, order versions, and per-symbol match IDs are separate values.

On the private order socket, check every sequenced subscription frame, including unknown frame kinds. Frames outside that sequence carry `seq` zero. Verify each unit's `prevCursor`, apply the whole unit, then persist `nextCursor`; also persist cursors from `snapshotEnd` and `covered`. The [private WebSocket guide](https://testnet.polyester.com/docs/developer-docs/connectrpc/private-websocket) owns the frame protocol, token renewal, and reconnect backoff.

`unavailable` is a freshness notice, not a dropped subscription. Keep the named orders, or the whole view when no orders are named, marked stale until `available`. The subscription recovers itself. `ready` or `synced` during that interval does not make stale state current.

## Refetch in this order

Restore authenticated subscriptions first and buffer live updates while reading. Stage snapshot pages and install them together at `snapshotEnd`. Keep consuming frames throughout recovery so you do not create another gap. Use the same account or Subaccount scope on every call.

The following sequence is a bot recovery procedure. These reads do not form an atomic snapshot across orders, fills, balances, and triggers.

1. Load every open order

   Call [`GetOpenOrders`](https://testnet.polyester.com/docs/api-docs/connect/orders.v1.OrdersReadService/GetOpenOrders) ([`GET /v1/orders/open`](https://testnet.polyester.com/docs/api-docs/rest/GET/v1/orders/open)). Include every Pair and side the bot manages, and page until the next token is empty. Request both attached-risk policy and state if the bot manages those legs.

   Retry `UNAVAILABLE` with the same read. An order missing from this list may have ended; absence does not prove a lost create failed or a cancel succeeded.

2. Load recent terminal history

   Call [`GetOrderHistory`](https://testnet.polyester.com/docs/api-docs/connect/orders.v1.OrdersReadService/GetOrderHistory) ([`GET /v1/orders/history`](https://testnet.polyester.com/docs/api-docs/rest/GET/v1/orders/history)) from at or before the last reconciled time, covering the outage. Read all pages, including after a short or empty page that still has a token. Keep scope, filters, and time bounds fixed for that traversal.

   Read each previously tracked order missing from the open list with [`GetOrder`](https://testnet.polyester.com/docs/api-docs/connect/orders.v1.OrdersReadService/GetOrder), even if it is outside the recent history range. Keep unresolved orders in your journal until their outcome is known.

   Merge by `orderId`, keeping the higher `version`. Track stream membership separately from your archive: `removedOrderIds` means an order left the live view, not that it was canceled. Do not erase attached-risk data merely because a read did not request it.

3. Backfill fills

   Call [`GetUserTrades`](https://testnet.polyester.com/docs/api-docs/connect/orders.v1.OrdersReadService/GetUserTrades) ([`GET /v1/spot/trades`](https://testnet.polyester.com/docs/api-docs/rest/GET/v1/spot/trades)) over an overlapping range and page to completion. Deduplicate by account, symbol, `matchId`, and `orderId`, including overlap with buffered live fills. A per-symbol `afterMatchId` must come from reconciled history, not just the highest ID observed live.

   History can lag order state. Re-read affected orders with execution history and check `executionsComplete` across their execution pages. Revisit incomplete fills, fees, and transfers; an empty page is not a reconciliation watermark. See [Fills and trade history](https://testnet.polyester.com/docs/developer-docs/shared-concepts/fills-and-trade-history).

4. Refresh balances

   Call [`GetBalances`](https://testnet.polyester.com/docs/api-docs/connect/ledger.read.v1.LedgerReadService/GetBalances) ([`GET /v1/balances`](https://testnet.polyester.com/docs/api-docs/rest/GET/v1/balances)). Merge buffered [balance updates](https://testnet.polyester.com/docs/api-docs/streams/private-ledger-balances) using `tradingRevision` for `trading`, `reserved`, and `available` together, and `fundingRevision` independently for `funding`. Keep the higher revision. Use returned `available` for sizing; see [Balances and holds](https://testnet.polyester.com/docs/developer-docs/account/balances-and-holds).

5. Refresh triggers and their children

   Page through [`ListTriggers`](https://testnet.polyester.com/docs/api-docs/connect/triggers.v1.TriggersService/ListTriggers) ([`GET /v1/triggers`](https://testnet.polyester.com/docs/api-docs/rest/GET/v1/triggers)), including triggers that ended during the outage. Filtering only active triggers would hide a stop that fired while disconnected.

   Backfill missing events with [`ListTriggerEvents`](https://testnet.polyester.com/docs/api-docs/connect/triggers.v1.TriggersService/ListTriggerEvents) ([`GET /v1/triggers/{trigger_id}/events`](https://testnet.polyester.com/docs/api-docs/rest/GET/v1/triggers/trigger_id/events)). Read newly discovered child orders and their fills. Deduplicate [trigger events](https://testnet.polyester.com/docs/api-docs/streams/private-spot-trigger-events) by `triggerId`, `eventType`, `childSeq`, and `tsNs` within the account. Treat buffered [trigger state updates](https://testnet.polyester.com/docs/api-docs/streams/private-spot-triggers) as a reason to refresh an affected trigger again, avoiding an older buffered view overwriting a later read.

After the reads, merge buffered updates and resolve every outstanding intent. Send `sync` on the private order subscription and wait for the matching `synced` after applying preceding units. This covers earlier order acknowledgements and reads; it is not a barrier for fee or balance settlement. Repeat affected reads if updates arrived during reconciliation or state remains stale.

Keep history enrichment, such as `feeTotal`, separate from the live order-state cache: stream orders omit it. Continue accounting backfills even after quoting resumes when execution state, available funds, and your strategy's risk checks permit it.

## Resolve outcome-unknown orders

Persist a [client order ID](https://testnet.polyester.com/docs/developer-docs/shared-concepts/client-order-ids) and the exact intent before the first create. On `ORDER_CREATE_OUTCOME_UNKNOWN`, a transport timeout, or a lost reply, retain that intent and ID. An immediate lookup miss does not prove non-admission.

Use [`GetOrder`](https://testnet.polyester.com/docs/api-docs/connect/orders.v1.OrdersReadService/GetOrder) with `clientOrderId` when the assigned order ID is unknown. REST single-order lookup accepts only `orderId`: page open orders and recent history and match `clientOrderId`. Retry temporary read unavailability with backoff. Once found, record the physical ID and reconcile its current state and fills.

A single-order create is not an exact replay. Reusing a retained client ID returns `CONFLICT_DUPLICATE_CLIENT_ORDER_ID`, even for the same payload; that conflict does not establish the earlier outcome. Do not change the ID and resubmit an unresolved intent. After a long outage, the reservation may have ended, so even the original ID cannot make resubmission safe by itself.

For `ORDER_CANCEL_OUTCOME_UNKNOWN` or `ORDER_MODIFY_OUTCOME_UNKNOWN`, refetch the affected orders before deciding whether another command is needed. Preserve any replayable request ID and payload when retrying the same command. Follow the operation's own replay rules, not the single-create rules. See [Error responses](https://testnet.polyester.com/docs/developer-docs/error-handling/error-responses) and [Error codes](https://testnet.polyester.com/docs/developer-docs/error-handling/error-codes).

## Reconcile the dead-man switch

The [dead-man switch](https://testnet.polyester.com/docs/developer-docs/shared-concepts/dead-man-switch) responds to `CancelAllAfter` heartbeats, not socket liveness. Reconnecting does not refresh it. If its earlier deadline passed, refetch orders even when a new heartbeat succeeds: cancellation may already be underway, and refreshing or disabling the switch does not undo it.

> **Triggers can still place orders**
>
> Expiry cancels scoped open orders, including trigger children, but never the triggers themselves. To stop automated trading completely, cancel live triggers first, then cancel orders, following [Cancel all orders](https://testnet.polyester.com/docs/developer-docs/shared-concepts/cancel-all-orders).

Choose deliberately whether your bot keeps heartbeating during recovery or lets its protection expire. Do not let a healthy socket alone drive heartbeats. Before requoting, arm the intended scope and confirm `effectiveScope` and the new deadline. Use a new `requestId` for each new heartbeat; retain it only for retries of that same call. If arming fails, keep quoting paused. Respect [rate limits](https://testnet.polyester.com/docs/developer-docs/shared-concepts/rate-limits) during recovery.

## Compact recovery loop

This Python-style pseudocode uses application-owned adapters, not SDK method names. `pages` follows every token; `merge_buffer` applies whole order units, deduplicates events, and compares balance revisions. It refreshes triggers invalidated during reads. Retry helpers back off with jitter and honor returned retry guidance.

```python
async def recover(bot):
    bot.pause_new_orders()
    checkpoint = bot.load_durable_checkpoint()
    # Subscriptions buffer continuously; cursors require matching local state.
    async with bot.restore_streams(checkpoint) as streams:
        while True:
            window = bot.overlapping_history_window(checkpoint)
            opened = await bot.pages("GetOpenOrders")
            ended = await bot.pages("GetOrderHistory", window=window)
            fills = await bot.pages("GetUserTrades", window=window)
            balances = await bot.read("GetBalances")
            triggers = await bot.pages("ListTriggers")
            trigger_events = await bot.backfill_trigger_events(triggers)

            bot.stage_reads(opened, ended, fills, balances, triggers)
            bot.stage_trigger_events(trigger_events)
            await bot.reconcile_new_children_and_unknown_intents()
            await bot.refresh_incomplete_execution_history()
            await streams.sync_orders_and_apply_through_synced()
            await bot.merge_buffer(streams)

            if bot.has_gap_or_stale_state_or_unresolved_risk():
                await bot.recovery_backoff()
                continue
            # Commit data and only the checkpoints it actually covers.
            bot.commit_state_and_checkpoints_atomically()
            # The heartbeat can start cancellation if the old timer expired.
            if await bot.arm_switch_and_confirm_scope():
                await bot.verify_orders_after_possible_expiry()
                await bot.merge_buffer(streams)
                if bot.can_resume_quoting():
                    bot.resume_new_orders()
                    break
            await bot.recovery_backoff()
        await bot.run_live(streams)  # Continue renewal, checks, and backfills.
```

Recovery succeeds when stream continuity is restored, stale views are cleared, unresolved intents cannot duplicate exposure, and the bot has reconciled orders, inventory, available funds, and trigger children. Keep late history and settlement reconciliation running in the live loop.