Custom Post-Processors

A post-processor converts a model's output tensors into results. The bundled post-processors are instances of IPostProcessor, a C structure of function pointers defined in NeuroN6_IPostProcessor.h. A sketch-defined instance is used by Vision in the same way.

#Interface

struct IPostProcessor {
  int32_t (*init)(void *ctx, NN_Instance_TypeDef *model);   // once, at begin
  int32_t (*run)(void *ctx, NN_Instance_TypeDef *model);    // after each inference
  void    (*draw_lcd)(void *ctx, OVL_Mode_t mode);          // overlay drawing, or NULL
  void    (*emit_metadata)(void *ctx);                      // host publishing, or NULL
  void    *ctx;                                             // implementation state
};

Vision.run() calls run, then draw_lcd with the overlay mode, then emit_metadata. Either of the last two may be NULL.

#Skeleton

A classifier with a single output tensor of class scores:

#include <OV5640_Arduino.h>
#include <PostProcess.h>
#pragma neuron6 model="my_classifier.tflite" name=my_net

OV5640 camera;
NEURON6_DECLARE_MODEL(my_net);

struct MyState {
  int   best_class;
  float best_score;
};
static MyState g_state;

static int32_t my_init(void *ctx, NN_Instance_TypeDef *m) {
  (void)ctx; (void)m;
  return 0;
}

static int32_t my_run(void *ctx, NN_Instance_TypeDef *m) {
  MyState *s = (MyState *)ctx;
  const LL_Buffer_InfoTypeDef *out = LL_ATON_Output_Buffers_Info(m);

  const int8_t *scores = (const int8_t *)LL_Buffer_addr_start(&out[0]);
  const uint32_t n     = LL_Buffer_len(&out[0]);
  const float scale    = out[0].scale[0];
  const int   zp       = out[0].offset[0];

  s->best_class = -1;
  s->best_score = 0.0f;
  for (uint32_t i = 0; i < n; i++) {
    float v = (scores[i] - zp) * scale;
    if (v > s->best_score) { s->best_score = v; s->best_class = (int)i; }
  }
  return 0;
}

static void my_draw_lcd(void *ctx, OVL_Mode_t mode) {
  MyState *s = (MyState *)ctx;
  uint16_t *ovl = OVL_GetBackBuffer();
  if (!ovl) return;
  ohm_lcd_bind_argb4444(ovl, 800, 480, 800);
  if (mode == OVL_CLEAR || mode == OVL_RELOAD_CLEAR) ohm_lcd_clear_transparent();
  ohm_lcd_set_font(&Font24);
  ohm_lcd_set_color(255, 255, 255);
  char line[48];
  snprintf(line, sizeof line, "class %d  %.0f%%", s->best_class, s->best_score * 100.0f);
  ohm_lcd_text(20, 20, line);
  if (mode == OVL_RELOAD || mode == OVL_RELOAD_CLEAR) OVL_Commit();
}

static void my_emit(void *ctx) {
  MyState *s = (MyState *)ctx;
  char line[48];
  snprintf(line, sizeof line, "class %d %.2f", s->best_class, s->best_score);
  vnd_meta_publish_debug_text(line);
}

static IPostProcessor g_my_pp = {
  .init = my_init, .run = my_run,
  .draw_lcd = my_draw_lcd, .emit_metadata = my_emit,
  .ctx = &g_state,
};

void setup() {
  VisionConfig cfg;
  cfg.nn.width = 224; cfg.nn.height = 224;
  cfg.nn.aspect = ASPECT_CROP_CENTER;
  Vision.begin(&NN_Instance_my_net, &g_my_pp, cfg);
}

void loop() {
  Vision.run();
}

#Tensor access

LL_ATON_Output_Buffers_Info(model) returns an array of output descriptors terminated by an entry with a NULL name. LL_Buffer_addr_start() returns the data pointer and LL_Buffer_len() the length in bytes. Each descriptor carries the shape, quantisation scale and zero point. A value is dequantised as (raw - zero_point) * scale.

Tensor names are assigned by the compiler and change between exports. Outputs are identified by shape, and where two share a shape, by content magnitude or order. Raw NPU Output shows a sketch that prints every output's shape.

#Coordinates

Positions reported by a model are normalised to its input, which under a crop is not the full frame. The helpers nn_norm_to_lcd_x() and nn_norm_to_cam_norm_x(), and their y counterparts, convert to panel pixels and camera-normalised coordinates respectively. They read the live crop and are correct after runRoi(). See Aspect Modes and Coordinates.

#Publishing

Three mechanisms send results to Neuro Studio.

Debug text. vnd_meta_publish_debug_text(line) shows a line in the terminal and is recorded in the boot log.

Vision publisher. For results with a bounding box, vnd_meta_vision_begin, vnd_meta_vision_add and vnd_meta_vision_finish draw the box on the video, label it and send a crop thumbnail to the Vision panel:

vnd_meta_vision_t vis;
vnd_meta_vision_begin(&vis, VND_META_SLOT_MODEL_0, META_VISION_GENERIC, payload, sizeof payload);
vnd_meta_vision_add(&vis, cx, cy, w, h, "label", extra, extra_len, snapshot, W, H);
vnd_meta_vision_finish(&vis);

Boxes are given in NN-normalised coordinates and mapped by the helper. finish() is called on every frame, including frames with no results, because an empty frame is what clears the previous box. A box without a label carries only a class index, which the host resolves through the shared class table; index 0 is "person".

Raw TLVs. vnd_meta_publish_tlv(slot, tag, data, len) publishes arbitrary bytes under a tag. The wire format is documented in n6_host_contract.h and vnd_meta.h.

#Overlay

The mode passed to draw_lcd indicates whether to clear before drawing and whether to commit after. Honouring it allows the post-processor to compose with others in a multi-model sketch. Drawing functions are described in The On-board Display. The _white drawing variants are used for output that must not be recoloured by another drawer in the same frame.

#Multi-model use

A custom post-processor works with runWith() and runRoi(). A factory that takes the slot as a parameter, as the bundled _OnSlot variants do, allows several instances to publish without collision. Slot 0 carries the frame-wide box list.

#Reference implementations

The source of every bundled post-processor is in libraries/PostProcess/src. The Nia decoder is a compact example: six tensors, per-tensor dequantisation, sigmoid, box decode and local maximum deduplication. The palm detector wrapper shows adaptation of a vendor decoder.