# Schemas and codecs Generate SBE codecs for the Polyester trading WebSocket and encode messages byte for byte: header, version, nulls, scaled integers, groups, and variable-length data. Every frame on the SBE WebSocket is one message of the trading schema: an 8-byte header, a fixed block, then any repeating groups and variable-length fields. Generate your codec from the schema of the environment you connect to, and let it handle offsets and null values. ## The schema The schema ID is `7`. Each environment publishes its own XML and accepts exactly one version of it. [Schema version](https://testnet.polyester.com/docs/developer-docs/sbe-websocket/getting-started#schema-version) links the published files and a download script that checks the XML against the manifest. Version rules: - Send the XML's `version` in every message header. Any other version is answered with a `CommandReject` with code `BAD_REQUEST` and is not processed. - The server encodes its messages with the same version, so decode with the same codec. - A new version can change any message. Regenerate your codec whenever the environment's manifest names a new schema file, before that version goes live on your environment. - A field's `sinceVersion` records when it was added. It does not let you send an older layout. > **Watch the manifest** > > A release can move Testnet to a new schema version. Pin your build to the Testnet manifest and check it as part of each deployment. ## Generate codecs Real Logic's [Simple Binary Encoding](https://github.com/aeron-io/simple-binary-encoding) tool generates codecs from the XML. Download the `sbe-all` jar of a recent release from Maven Central (`uk.co.real-logic:sbe-all`) and run it with Java 17 or later. Pin the release you use so that regenerated code stays stable. ## Java ```bash java --add-opens java.base/jdk.internal.misc=ALL-UNNAMED \ -Dsbe.output.dir=generated/java \ -jar sbe-all-.jar tradews.xml ``` Java is the default target. The codecs land in the `tradews` package and use the Agrona library for their buffers. ## C++ ```bash java --add-opens java.base/jdk.internal.misc=ALL-UNNAMED \ -Dsbe.target.language=Cpp \ -Dsbe.output.dir=generated/cpp \ -jar sbe-all-.jar tradews.xml ``` The C++ codecs are header-only. ## Go ```bash java --add-opens java.base/jdk.internal.misc=ALL-UNNAMED \ -Dsbe.target.language=Golang \ -Dsbe.go.generate.generate.flyweights=true \ -Dsbe.output.dir=generated/go \ -jar sbe-all-.jar tradews.xml ``` Flyweights encode and decode in place without allocating per message. Without the flag, the Go generator emits struct codecs instead. ## Rust ```bash java --add-opens java.base/jdk.internal.misc=ALL-UNNAMED \ -Dsbe.target.language=Rust \ -Dsbe.output.dir=generated/rust \ -jar sbe-all-.jar tradews.xml ``` The output is a crate you can add to your workspace as a path dependency. ## C\# ```bash java --add-opens java.base/jdk.internal.misc=ALL-UNNAMED \ -Dsbe.target.language=uk.co.real_logic.sbe.generation.csharp.CSharp \ -Dsbe.output.dir=generated/csharp \ -jar sbe-all-.jar tradews.xml ``` The generated code needs the SBE C# runtime library; see the [C# user guide](https://github.com/aeron-io/simple-binary-encoding/wiki/CSharp-User-Guide). ## Other languages Python, TypeScript, and other languages have no generator from Real Logic. The layout is small and fixed, so a hand-written codec over `struct`, `DataView`, or your language's byte buffer works well. Take offsets and null values from the [Message reference](https://testnet.polyester.com/docs/developer-docs/sbe-websocket/message-reference), and test your encoder against the worked examples below. The [Getting started quickstart](https://testnet.polyester.com/docs/developer-docs/sbe-websocket/getting-started#quickstart) shows a complete Python client written this way. ## Message layout ```text +----------------------+--------------------+----------------------+--------------------------+ | messageHeader 8 B | fixed block | repeating groups | variable-length fields | | blockLength | blockLength bytes | 4 B size header + | 4 B length + bytes, | | templateId | | entries, in schema | in schema order | | schemaId = 7 | | order | | | version | | | | +----------------------+--------------------+----------------------+--------------------------+ ``` The rules every message follows: - **Byte order:** every integer is little-endian. - **Header:** four `uint16` values, `blockLength`, `templateId`, `schemaId`, and `version`. The `templateId` selects the message; see the [template index](https://testnet.polyester.com/docs/developer-docs/sbe-websocket/message-reference#template-index). - **Fixed block:** fields sit at fixed offsets with no padding. `blockLength` must equal the block length of that message in your environment's schema. A different value is rejected. - **Groups:** a 4-byte header of `uint16` entry block length and `uint16` entry count, then the entries. Each entry is its own fixed block followed by its own variable-length fields. - **Variable-length fields:** a `uint32` length, then that many bytes. An empty value is a length of `0`. Note the 4-byte length: many SBE examples use a 2-byte length. - **Order:** write and read the parts in schema order. Generated codecs enforce this. - **Trailing bytes:** a frame must end exactly at its last field. Extra bytes are rejected. When you decode server messages, read `blockLength` from the header and skip to the end of the fixed block by that value, as generated codecs do. ## Field encodings ### Null values Optional fields carry a null value when they are not set. The schema uses two conventions: - **Optional `int64`** · null `-9223372036854775808` (`INT64_MIN`)\ For example `baseQtyScaled`, `priceTicks`, `expireAtTsNs`. - **Optional `int32`** · null `-2147483648` (`INT32_MIN`)\ For example `marketMaxSlippageBps`. - **Optional `uint64`** · null `18446744073709551615` (`UINT64_MAX`)\ For example `rateLimitRemaining`. - **IDs and enums of a `ZeroNull` type** · null `0`\ For example `orderId`, `symbolId` in a cancel, `side`, `stpMode`, `RejectCode`. Optional IDs and enums use `0` as their null value. For example, a cancel by client order ID sets `orderId` to `0`, and a `symbolId` of `0` lets the service resolve the Pair from the order. A required enum whose type is zero-null, such as `side`, rejects `0`. A required enum with a `uint8` encoding, such as `orderType`, has no value `0` either. > **Zero is a value for int64 fields** > > An unset `int64` field must hold `INT64_MIN`, not `0`. A zero-initialized buffer turns every unset price or quantity into an explicit `0`: a `MARKET` order with `priceTicks` `0` is rejected, and so is a `MAX_QUOTE_DEBIT` order whose `baseQtyScaled` is `0` instead of null. Set every optional field explicitly, or start from a buffer your codec has filled with null values. ### Prices, quantities, and timestamps | Field pattern | Encoding | | ----------------------------- | -------------------------------------------------------- | | `*PriceTicks`, `priceTicks` | Price in quote units scaled by 109, for every Pair | | `marketMaxSlippageTicks` | Price delta in the same 10-9 quote-unit ticks | | `*QtyScaled`, `baseQtyScaled` | Base quantity scaled by the Pair's `base_quantity_scale` | | `maxQuoteDebitScaled` | Quote amount scaled by the Pair's `quote_quantity_scale` | | `*Bps` | Basis points: `1` is 0.01% | | `*TsNs` | Nanoseconds since the Unix epoch, UTC | | `*Ns` durations | Nanoseconds | Read each Pair's `symbol_id`, `base_quantity_scale`, `quote_quantity_scale`, tick size, and step size from [`GET /v1/spot/config`](https://testnet.polyester.com/docs/api-docs/rest/GET/v1/spot/config) (ConnectRPC [`GetSpotConfig`](https://testnet.polyester.com/docs/api-docs/connect/marketdata.v1.MarketDataService/GetSpotConfig)). Price ticks never use the quote Asset's quantity scale. See [Scaled integers](https://testnet.polyester.com/docs/developer-docs/connectrpc/scaled-integers) for conversion rules and common mistakes. ### Identifiers `orderId` fields carry the public order ID as an unsigned 64-bit integer, the same value ConnectRPC carries as `fixed64`. REST shows the same ID as a base58 string. See [ConnectRPC identifiers](https://testnet.polyester.com/docs/developer-docs/connectrpc/identifiers) and [Public IDs](https://testnet.polyester.com/docs/developer-docs/shared-concepts/public-ids). `symbolId` is the Pair's numeric `symbol_id`. `clientOrderId` and `newClientOrderId` are ASCII: 1 to 36 letters, digits, `.`, `_`, `:`, `/`, or `-`. See [Client order IDs](https://testnet.polyester.com/docs/developer-docs/shared-concepts/client-order-ids). The `detail` text of a `CommandReject` is UTF-8. ### Enums and bitsets Enums are one or two bytes; their values are listed in the [Message reference](https://testnet.polyester.com/docs/developer-docs/sbe-websocket/message-reference#enums-and-bitsets). The two bitsets are `uint32`: in `EventChannels`, bit 0 is `executions`, bit 1 `orders`, and bit 2 `marketResync`; in `OrderFlags`, bit 0 is `postOnly`. Every other bit must be `0`. Boolean fields such as `dryRun` and `cancelledOrdersTruncated` are `uint8` values `0` or `1`. ## Worked examples The examples are real encodings, checked against codecs generated from the schema. `vv vv` stands for your environment's schema version as a little-endian `uint16`. Offsets count from the start of the frame. A `Ping` with `seq` `1`, `correlationId` `100`, and payload `hello` is 33 bytes: ```text 0 10 00 blockLength = 16 2 01 00 templateId = 1 (Ping) 4 07 00 schemaId = 7 6 vv vv version 8 01 00 00 00 00 00 00 00 seq = 1 16 64 00 00 00 00 00 00 00 correlationId = 100 24 05 00 00 00 payload length = 5 28 68 65 6c 6c 6f payload = "hello" ``` The server answers with a `Pong` of the same shape, template ID `22`, echoing `seq`, `correlationId`, and payload. ### CreateOrder: post-only limit sell, 106 bytes Example values: Pair `symbolId` `7`, a quantity of `0.01` on a Pair with `base_quantity_scale` `8`, a price of `65000.5`, and client order ID `quickstart-1`. ```text 0 52 00 blockLength = 82 2 02 00 templateId = 2 (CreateOrder) 4 07 00 schemaId = 7 6 vv vv version 8 03 00 00 00 00 00 00 00 seq = 3 16 66 00 00 00 00 00 00 00 correlationId = 102 24 07 00 00 00 symbolId = 7 28 02 side = 2 (SELL) 29 01 orderType = 1 (LIMIT) 30 01 timeInForce = 1 (GTC) 31 40 42 0f 00 00 00 00 00 baseQtyScaled = 1000000 39 00 75 b6 1b 1e 3b 00 00 priceTicks = 65000500000000 47 01 00 00 00 flags = 0x1 (postOnly) 51 01 feeAsset = 1 (QUOTE) 52 01 stpMode = 1 (EXPIRE_MAKER) 53 00 00 00 00 00 00 00 80 marketClientRefPriceTicks = null 61 00 00 00 00 00 00 00 80 marketMaxSlippageTicks = null 69 00 00 00 80 marketMaxSlippageBps = null 73 01 sizingKind = 1 (BASE_QUANTITY) 74 00 00 00 00 00 00 00 80 maxQuoteDebitScaled = null 82 00 00 00 00 00 00 00 80 expireAtTsNs = null 90 0c 00 00 00 clientOrderId length = 12 94 71 75 69 63 6b 73 74 61 72 74 2d 31 clientOrderId = "quickstart-1" ``` ### CancelOrder by client order ID, 52 bytes ```text 0 1c 00 blockLength = 28 2 03 00 templateId = 3 (CancelOrder) 4 07 00 schemaId = 7 6 vv vv version 8 04 00 00 00 00 00 00 00 seq = 4 16 67 00 00 00 00 00 00 00 correlationId = 103 24 00 00 00 00 00 00 00 00 orderId = 0 (null: cancel by client order ID) 32 07 00 00 00 symbolId = 7 36 0c 00 00 00 clientOrderId length = 12 40 71 75 69 63 6b 73 74 61 72 74 2d 31 clientOrderId = "quickstart-1" ``` ### Subscribe to executions and orders, 28 bytes ```text 0 14 00 blockLength = 20 2 06 00 templateId = 6 (Subscribe) 4 07 00 schemaId = 7 6 vv vv version 8 02 00 00 00 00 00 00 00 seq = 2 16 65 00 00 00 00 00 00 00 correlationId = 101 24 03 00 00 00 channels = 0x3 (executions, orders) ``` ## Test your codec - Encode a `Ping` and compare it byte for byte with the example above. - Send the `Ping` on Testnet and check that `Pong` echoes all three values. - Send a command with a deliberately wrong version and check that you decode the `CommandReject`, including its `detail` text. - Decode every server template, even ones you do not act on, so that an unexpected message never stalls your reader.