Recover after reconnect

Pause new quoting while you reconnect and reconcile account state. Orders can fill or end while your bot is offline, triggers can place children, and a lost command reply can leave its outcome unknown. Resume once the affected state is current and your pending intents are resolved.

Use https://api.testnet.polyester.com for reads and wss://api.testnet.polyester.com/v1/ws/private for private order updates. The trade socket uses the host trade.api.testnet.polyester.com. Testnet is the public environment; Mainnet is coming soon.

What can be lost

A reconnect can replay duplicate events or leave a gap. When the private order socket cannot resume, its fresh snapshot replaces order state but does not replay the intervening amend, cancel, or rejection outcomes. Fills remain available from trade history. The snapshot includes open, pending, and recently ended orders, not a complete archive.

Keep a durable journal of submitted intents, client order IDs, assigned order IDs, and replies. Save stream checkpoints atomically with the data they describe. A restart without the matching local state requires a fresh snapshot and history reconciliation, even if a cursor survived.

Detect a gap

The private WebSocket carries your orders, and the realtime connection in the WebSocket session model carries the event streams. Each signals lost state differently:

Order sequence: if private order seq skips or repeats, drop that subscription and resubscribe with the last applied cursor.
Order resync: on resyncRequired or a changed snapshot epoch, install the fresh snapshot and refetch history.
Realtime recovery: on recovered=false or a stream-position gap, backfill the affected stream through its read API.
Order version: when a read has a higher version, keep the newer state and reconcile any missing executions.

Order versions compare copies of one physical order: 0 means accepted but not yet on the book, then published changes increase the version. They are not account-wide sequence numbers. A jump can reveal missed intermediate states; equal versions do not prove you received every fill or command outcome. Stream positions, order versions, and per-symbol match IDs are separate values.

On the private order socket, check every sequenced subscription frame, including unknown frame kinds. Frames outside that sequence carry seq zero. Verify each unit's prevCursor, apply the whole unit, then persist nextCursor; also persist cursors from snapshotEnd and covered. The private WebSocket guide owns the frame protocol, token renewal, and reconnect backoff.

unavailable is a freshness notice, not a dropped subscription. Keep the named orders, or the whole view when no orders are named, marked stale until available. The subscription recovers itself. ready or synced during that interval does not make stale state current.

Refetch in this order

Restore authenticated subscriptions first and buffer live updates while reading. Stage snapshot pages and install them together at snapshotEnd. Keep consuming frames throughout recovery so you do not create another gap. Use the same account or Subaccount scope on every call.

The following sequence is a bot recovery procedure. These reads do not form an atomic snapshot across orders, fills, balances, and triggers.

Load every open order

Call GetOpenOrders (GET /v1/orders/open). Include every Pair and side the bot manages, and page until the next token is empty. Request both attached-risk policy and state if the bot manages those legs.

Retry UNAVAILABLE with the same read. An order missing from this list may have ended; absence does not prove a lost create failed or a cancel succeeded.

Load recent terminal history

Call GetOrderHistory (GET /v1/orders/history) from at or before the last reconciled time, covering the outage. Read all pages, including after a short or empty page that still has a token. Keep scope, filters, and time bounds fixed for that traversal.

Read each previously tracked order missing from the open list with GetOrder, even if it is outside the recent history range. Keep unresolved orders in your journal until their outcome is known.

Merge by orderId, keeping the higher version. Track stream membership separately from your archive: removedOrderIds means an order left the live view, not that it was canceled. Do not erase attached-risk data merely because a read did not request it.

Backfill fills

Call GetUserTrades (GET /v1/spot/trades) over an overlapping range and page to completion. Deduplicate by account, symbol, matchId, and orderId, including overlap with buffered live fills. A per-symbol afterMatchId must come from reconciled history, not just the highest ID observed live.

History can lag order state. Re-read affected orders with execution history and check executionsComplete across their execution pages. Revisit incomplete fills, fees, and transfers; an empty page is not a reconciliation watermark. See Fills and trade history.

Refresh balances

Call GetBalances (GET /v1/balances). Merge buffered balance updates using tradingRevision for trading, reserved, and available together, and fundingRevision independently for funding. Keep the higher revision. Use returned available for sizing; see Balances and holds.

Refresh triggers and their children

Page through ListTriggers (GET /v1/triggers), including triggers that ended during the outage. Filtering only active triggers would hide a stop that fired while disconnected.

Backfill missing events with ListTriggerEvents (GET /v1/triggers/{trigger_id}/events). Read newly discovered child orders and their fills. Deduplicate trigger events by triggerId, eventType, childSeq, and tsNs within the account. Treat buffered trigger state updates as a reason to refresh an affected trigger again, avoiding an older buffered view overwriting a later read.

After the reads, merge buffered updates and resolve every outstanding intent. Send sync on the private order subscription and wait for the matching synced after applying preceding units. This covers earlier order acknowledgements and reads; it is not a barrier for fee or balance settlement. Repeat affected reads if updates arrived during reconciliation or state remains stale.

Keep history enrichment, such as feeTotal, separate from the live order-state cache: stream orders omit it. Continue accounting backfills even after quoting resumes when execution state, available funds, and your strategy's risk checks permit it.

Resolve outcome-unknown orders

Persist a client order ID and the exact intent before the first create. On ORDER_CREATE_OUTCOME_UNKNOWN, a transport timeout, or a lost reply, retain that intent and ID. An immediate lookup miss does not prove non-admission.

Use GetOrder with clientOrderId when the assigned order ID is unknown. REST single-order lookup accepts only orderId: page open orders and recent history and match clientOrderId. Retry temporary read unavailability with backoff. Once found, record the physical ID and reconcile its current state and fills.

A single-order create is not an exact replay. Reusing a retained client ID returns CONFLICT_DUPLICATE_CLIENT_ORDER_ID, even for the same payload; that conflict does not establish the earlier outcome. Do not change the ID and resubmit an unresolved intent. After a long outage, the reservation may have ended, so even the original ID cannot make resubmission safe by itself.

For ORDER_CANCEL_OUTCOME_UNKNOWN or ORDER_MODIFY_OUTCOME_UNKNOWN, refetch the affected orders before deciding whether another command is needed. Preserve any replayable request ID and payload when retrying the same command. Follow the operation's own replay rules, not the single-create rules. See Error responses and Error codes.

Reconcile the dead-man switch

The dead-man switch responds to CancelAllAfter heartbeats, not socket liveness. Reconnecting does not refresh it. If its earlier deadline passed, refetch orders even when a new heartbeat succeeds: cancellation may already be underway, and refreshing or disabling the switch does not undo it.

Triggers can still place orders

Expiry cancels scoped open orders, including trigger children, but never the triggers themselves. To stop automated trading completely, cancel live triggers first, then cancel orders, following Cancel all orders.

Choose deliberately whether your bot keeps heartbeating during recovery or lets its protection expire. Do not let a healthy socket alone drive heartbeats. Before requoting, arm the intended scope and confirm effectiveScope and the new deadline. Use a new requestId for each new heartbeat; retain it only for retries of that same call. If arming fails, keep quoting paused. Respect rate limits during recovery.

Compact recovery loop

This Python-style pseudocode uses application-owned adapters, not SDK method names. pages follows every token; merge_buffer applies whole order units, deduplicates events, and compares balance revisions. It refreshes triggers invalidated during reads. Retry helpers back off with jitter and honor returned retry guidance.

async def recover(bot):
    bot.pause_new_orders()
    checkpoint = bot.load_durable_checkpoint()
    # Subscriptions buffer continuously; cursors require matching local state.
    async with bot.restore_streams(checkpoint) as streams:
        while True:
            window = bot.overlapping_history_window(checkpoint)
            opened = await bot.pages("GetOpenOrders")
            ended = await bot.pages("GetOrderHistory", window=window)
            fills = await bot.pages("GetUserTrades", window=window)
            balances = await bot.read("GetBalances")
            triggers = await bot.pages("ListTriggers")
            trigger_events = await bot.backfill_trigger_events(triggers)

            bot.stage_reads(opened, ended, fills, balances, triggers)
            bot.stage_trigger_events(trigger_events)
            await bot.reconcile_new_children_and_unknown_intents()
            await bot.refresh_incomplete_execution_history()
            await streams.sync_orders_and_apply_through_synced()
            await bot.merge_buffer(streams)

            if bot.has_gap_or_stale_state_or_unresolved_risk():
                await bot.recovery_backoff()
                continue
            # Commit data and only the checkpoints it actually covers.
            bot.commit_state_and_checkpoints_atomically()
            # The heartbeat can start cancellation if the old timer expired.
            if await bot.arm_switch_and_confirm_scope():
                await bot.verify_orders_after_possible_expiry()
                await bot.merge_buffer(streams)
                if bot.can_resume_quoting():
                    bot.resume_new_orders()
                    break
            await bot.recovery_backoff()
        await bot.run_live(streams)  # Continue renewal, checks, and backfills.

Recovery succeeds when stream continuity is restored, stale views are cleared, unresolved intents cannot duplicate exposure, and the bot has reconciled orders, inventory, available funds, and trigger children. Keep late history and settlement reconciliation running in the live loop.

Order types and lifecycle Order types, time in force, statuses and their transitions, cancel and reject reasons, versions, modify and replace, and how to read orders and fills. Fills and trade history Read your executions, reconcile order totals and fees, and backfill a durable trading history. Order sizing Choose exact base-quantity or maximum quote-debit sizing and read the authoritative quantity accepted for execution. Client order IDs and idempotency Use client order IDs to prevent duplicate placement, correlate client intents, and reconcile uncertain outcomes. Preview order Check whether an order is currently admissible without creating an order, reserving funds, or presenting an execution quote. Market order price protection Understand market-order slippage overrides, Pair defaults, client reference drift, execution boundaries, and stale-quote errors. Self-trade prevention Prevent an incoming order from trading against a resting order of the same Main Account or its Subaccounts. Batch orders Create, replace, or cancel many orders in one request, read each item's result, and retry safely after a partial failure or an unknown outcome. Cancel all orders Cancel every open order that matches a filter, with an answer at acceptance and idempotent retries. Dead-man switch Arm a heartbeat timer with CancelAllAfter so your open orders are canceled automatically if your system stops refreshing it. Fee Assets Choose whether a BUY fee is charged in quote or deducted from received base, and decode fee amounts with the correct scale. Swap Convert Trading balances through spot order books with quotes, minimum output protection, and idempotent execution. Triggers overview How standalone and attached triggers watch a price or a schedule, place child orders, report their status, and count against limits. Rate limits Weighted quotas, default limits, rejection metadata, and retry behavior. WebSocket session model Understand shared connection, authentication, subscription, and recovery behavior for Polyester realtime channels. Public IDs Understand what Polyester public IDs identify, why they are opaque, and how their representation differs per interface.