Publishing Data from a Sketch

A sketch sends data to Neuro Studio by publishing metadata records into slots, which the core attaches to the next video frame or sends on their own when there is no camera. The bundled post-processors and sensor libraries do this internally. This page covers the functions available to a sketch that produces its own results. All of them are in PostProcess.h.

#Text

vnd_meta_publish_debug_text("[app] calibration loaded");

Publishes a line to the Terminal panel. Lines published before the host connects are kept in a 2 KB log and replayed on connection. Serial.print output reaches the terminal too, but is not captured in the boot log.

#Sensor readings

The bundled sensor libraries publish through sendNS():

imu.sendNS();
mag.sendNS();
tof.sendNS();

Each writes its slot. With a camera the readings ride on every frame; without one the core emits them as metadata-only frames. A publisher that must not stall behind a long I2C claim uses n6_i2c_trylock() first.

#Results with a box

The vision publisher draws a box on the video, labels it, sends extra bytes to the Vision panel, and provides a crop thumbnail when a host is subscribed:

static uint8_t g_payload[4096] __attribute__((section(".ext_psram")));

vnd_meta_vision_t vis;
vnd_meta_vision_begin(&vis, VND_META_SLOT_QR, META_QR_FRAME, g_payload, sizeof g_payload);

// per result; box in normalised coordinates of the analysed window
vnd_meta_vision_add(&vis, cx, cy, w, h, "label", extra, extra_len, g_snapshot, W, H);

vnd_meta_vision_finish(&vis);

begin names the slot, the record type and a working buffer. add appends one result: a box, a label of up to VND_VISION_LABEL_MAX characters, an optional block of extra bytes shown in the panel, and the image the crop is cut from. finish publishes. Three rules apply:

• finish() is called on every frame, including frames with no results. A record with a count of zero is what clears the previous box from the video and the panel.
• Boxes are given in the coordinates of the window that was analysed, normalised 0 to 1, and are mapped to the full frame by the helper.
• A box without a label carries only a class index, which the host resolves through the shared class table. Index 0 is "person".

vnd_meta_vision_announce(kind) is called about once per second with a bitmask of the detectors the sketch runs (bit 0 QR, bit 1 barcode, bit 2 blob, bit 3 AprilTag), so the panel titles itself. The crop is sent only when the host has subscribed to evidence, and the call costs nothing otherwise.

The QR, barcode, blob and AprilTag examples use this path. Their extra bytes follow the layouts in Wire Protocol.

#Custom records

For anything else, a record of any type is published into a slot:

uint8_t payload[8];
vnd_meta_put_u32_le(&payload[0], counter);
vnd_meta_put_u32_le(&payload[4], millis());
vnd_meta_publish_tlv(VND_META_SLOT_MODEL_3, 0x4F, payload, sizeof payload);

A type not known to Neuro Studio is skipped by length, so a custom record needs a matching decoder in the host before it is displayed. New evidence types are allocated in the range 0x70 to 0x7F; other new types are given values not already listed in vnd_meta.h. The contract header's five-question procedure decides whether a new record is a capability, a result or evidence.

vnd_meta_publish_tlv has a weak stub in the core, so a library can call it without depending on PostProcess.

The builder functions assemble a record from parts when it is too large to stage in one buffer:

vnd_meta_builder_t b;
vnd_meta_builder_begin(&b, VND_META_SLOT_MODEL_3);
vnd_meta_builder_put_tlv(&b, 0x4F, payload, sizeof payload);
vnd_meta_builder_finish(&b);

Helpers vnd_meta_put_u16_le, vnd_meta_put_u32_le, vnd_meta_f01_to_u16 (a 0 to 1 float to the wire's u16) and vnd_meta_conf_to_q8 (a confidence to one byte) produce the field encodings the host expects.

#Slots

A slot holds one record set and keeps it until rewritten. vnd_meta_invalidate_slot(slot) clears it, which is required whenever a producer stops publishing so that its last result does not remain on screen. A custom result normally uses an unused model slot.

#Announcements without video

NeuroN6_MetaAnnounce() publishes a record and repeats it at about 1 Hz for a period, so that a board with no camera, or one whose host attaches late, still delivers it. The N6Events library uses it for events.

#Receiving commands

A sketch does not normally handle downlink packets; the core routes TUN0 to the tuning registry, CAM0 to the sensor, ISP0 to the ISP path, SUB0 to the evidence gate and the asset packets to the asset store. Live parameters are exposed by registering variables, described in Live Tuning and Assets.

#What travels and what does not

Results are computed for the board's own purposes and always published; the host overhears them. Evidence is produced only on request. Nothing a host sets through a parameter survives a reboot. These rules and the reasoning behind them are in The Host Contract.