Wire Protocol
The framed stream the Neuro N6 sends to Neuro Studio, the TLV metadata records, the downlink commands, and the evidence subscription mechanism.
The Neuro N6 sends Neuro Studio a stream of frames, each carrying a header, a block of metadata records and, usually, a JPEG image. The same byte stream is sent over the USB vendor bulk endpoint and over TCP on port 5000 for WiFi. Commands from the host travel back on the USB vendor bulk-out endpoint or the same TCP connection as small magic-prefixed packets. The format is defined in the core's libraries/PostProcess/src/vnd_meta.h and cores/NeuroN6/n6_host_contract.h, and decoded in Neuro Studio's mjpeg_serial_worker.dart and mjpeg_overlay.dart.
USB interfaces
The board enumerates as a composite device: interface 0 is a vendor interface with a bulk-out endpoint 0x01 and a bulk-in endpoint 0x81, and interfaces 1 and 2 are a CDC serial port. The device reports USB 2.01 and a Microsoft OS 2.0 descriptor set that names WinUSB as the driver for the vendor interface, so Windows 8.1 and later bind it without a driver installation. Neuro Studio reads the vendor interface through libusb. The identity is VID 0x1209, PID 0x8008.
Frame header
Each frame begins with a 20-byte little-endian header:
| Offset | Field | Meaning |
|---|---|---|
| 0 | magic | 0x30564E44, the bytes D N V 0 |
| 4 | frame_id | Incrementing counter |
| 8 | meta_len | Bytes of metadata that follow the header |
| 12 | payload_len | Bytes of image that follow the metadata |
| 16 | flags | Bit 0 set for JPEG |
The metadata block follows the header and the JPEG follows the metadata. A frame may carry metadata and no image, for example a sensor reading from a board with no camera or the periodic address announcement; payload_len is then zero. The host parser scans for the magic to resynchronise after corruption and discards the buffer if no frame parses within 8 MB.
Metadata records
The metadata block is a sequence of type-length-value records: a one-byte type, a two-byte little-endian length, and the value. A decoder skips records of unknown type by length, which is how older hosts tolerate newer firmware. Coordinates are 16-bit unsigned values scaled from 0 to 1 across the full camera frame; scores are one byte scaled from 0 to 255.
| Type | Name | Value |
|---|---|---|
0x01 | META_INFERENCE_MS | u16 milliseconds |
0x02 | META_BOXES | u8 count, then per box: u16 cx, cy, w, h; u8 score; u8 class |
0x03 | META_KEYPOINTS | u8 count, then per point: u8 id, u8 colour, u16 x, y, u8 confidence |
0x04 | META_BINDINGS | Pairs of keypoint ids to join with lines |
0x0A | META_ISEG_CLASSSTR | Class name strings by box index |
0x0B | META_ISEG_MASK | Compressed mask bitmap by box index |
0x12 | Warning | Text held on screen for a few seconds |
0x15, 0x16 | META_TOF_FRAME, status | ToF zone distances and statuses, with a 4-byte grid header |
0x17 | META_DEBUG_TEXT | Terminal text |
0x1A | META_MAG_FRAME | 4 floats: x, y, z, temperature |
0x1B | META_IMU_FRAME | 7 floats: acceleration xyz, angular rate xyz, temperature |
0x20 | META_BOARD_IPV4 | 4 bytes address, u32 port |
0x30 | META_THERMAL_FRAME | u16 width, height; u8 format, flags; u16 reserved; width x height u16 pixels |
0x40 | META_VD66_ISP_ENABLED | u8 mode |
0x50 | META_AUDIO_PCM | u32 sample rate, u8 channels, u8 bits, u16 count, int16 samples |
0x51 | META_AUDIO_CLASS | u8 count, u8 top, u16 level, u16 threshold x1000, per class u16 score x1000 |
0x52 | META_AUDIO_MEL | u8 mels, u8 frames, u8 layout, u8 reserved, mel x frames bytes |
0x53 | META_AUDIO_LABELS | u8 count, then per class u8 length and characters |
0x60 | META_VISION_ID | u8 bitmask: bit 0 QR, bit 1 barcode, bit 2 blob, bit 3 AprilTag |
0x61 | META_QR_FRAME | u8 count, then per result: box, u8 version, u8 ECC level, u16 length, text |
0x62 | META_BARCODE_FRAME | u8 count, then per result: box, u8 symbology, u16 length, digits |
0x63 | META_BLOB_FRAME | u8 count, then box, u16 area, u16 hue, u16 pixels |
0x64 | META_CAM_CAPS | u8 bitmask: exposure, gain, white balance, autofocus |
0x65 | META_TUNE_DESC | u8 count, then per parameter: id, type, hint, name, i32 low, high, current |
0x66 | META_APRILTAG | u8 count, then per tag: box, u16 id, u8 family, u8 hamming, u16 margin, i16 angle, u8 pose valid, i16 tx, ty, tz in mm, i16 rx, ry, rz in centidegrees |
0x67 | META_EVENT | 17 bytes: u32 id, u32 time, u8 sources, u8 state, u16 score, u16 pre frames, u16 post frames, u8 sinks |
0x70 | META_EVIDENCE_CROP | u8 count, then per crop: u8 result index, u8 width, u8 height, RGB565 pixels |
Types 0x70 to 0x7F are reserved for evidence.
Slots and composition
On the board each producer writes into its own metadata slot: four for models, one each for ToF, IMU, magnetometer, thermal, debug text, QR, barcode, blob, vision identity, AprilTag, the three audio records, evidence and events. The capture task concatenates every ready slot in slot order into one metadata block per frame. A slot keeps its last contents until it is rewritten or invalidated, so a producer that stops publishing continues to be sent until vnd_meta_invalidate_slot() is called. Audio PCM is an ordered stream rather than a latest-wins slot and travels in frames of its own.
Traffic classes
The host contract divides uplink records into three classes. Capability announcements (0x20, 0x40, 0x53, 0x60, 0x64, 0x65) are republished about once per second and are safe to miss. Results are small, per frame and never dropped. Evidence (0x17, 0x52 and the 0x70 range) exists only for a person to look at, may be dropped under back-pressure, and is produced only while a host is subscribed. See The Host Contract.
Downlink commands
Commands are at most 64 bytes and begin with a four-character little-endian magic.
| Magic | Size | Function |
|---|---|---|
TUN0 | 10 | Set one live tuning parameter: u8 id, u8 type, i32 value (floats in thousandths) |
CAM0 | 12 | Capture controls: u8 flags for auto exposure, gain and white balance; u32 exposure in microseconds; u16 gain x256; u8 apply mask |
SUB0 | 8 | Evidence subscription: u32 mask, bit 0 for crop thumbnails. Zero unsubscribes |
ISP0 | 13 or more | ISP tuning for the VD66GY, parsed by the ISP path |
CFG0 | 16 | ToF configuration: profile, region size and step, timing budget, signal and sigma thresholds |
AST0, AST1, AST2 | 16, 7 plus data, 4 | Asset transfer: begin with length, CRC32 and version; chunk of up to 48 bytes at an offset; commit and reset |
The structures are fixed by static assertions in the contract header so that a field cannot be added without failing the build. Commands other than the asset transfer are not acknowledged.
Evidence subscription
Evidence is not sent unless a host asks for it. Neuro Studio sends SUB0 about once per second while it wants thumbnails; the subscription expires after 3 seconds without renewal, so a closed or crashed application stops the work within a few frames. Host presence alone is not used as the gate, because a board plugged in for power enumerates with nothing listening.
Asset transfer
An asset larger than one command is sent as AST0 with the total length, CRC32 and version, a sequence of AST1 chunks at explicit offsets so that a lost chunk is detected rather than shifting the rest, and AST2 to verify and commit. Committing resets the board, and the FSBL writes the flash. The camera calibration tool tools/push_calibration.py uses this path. See Live Tuning and Assets.
Network transport
Over WiFi the board's video task writes the identical framed stream to a TCP socket on port 5000 and reads commands from it. The stream is unauthenticated and unencrypted, and is intended for a point-to-point or trusted local network. While a USB connection is also present the board publishes its address in META_BOARD_IPV4 so the host can connect without configuration.
Bootloader protocol
The DFU bootloader uses a separate framed protocol over the CDC serial port, described in Bootloader Tool.