First Model

The ObjectDetection example, under File > Examples > NeuroN6Examples > 03.NeuralNetwork, runs an object detector on the NPU and streams labelled boxes to Neuro Studio. It is the simplest complete neural network sketch in the core.

#include <OV5640_Arduino.h>
#include <PostProcess.h>

#pragma neuron6 model="nia_1_int8.onnx" name=nia_v4

OV5640 camera;
NEURON6_DECLARE_MODEL(nia_v4);

IPostProcessor *nia_pp = nullptr;

void setup() {
  VisionConfig cfg;
  cfg.camera    = CAMERA_VGA;
  cfg.fps       = FPS_45;
  cfg.nn.width  = 256;              // model input size
  cfg.nn.height = 256;
  cfg.nn.aspect = ASPECT_STRETCH;
  cfg.transport = TRANSPORT_USB;

  nia_pp = PostProcess_NIA_OnSlot(0.25f, 0.60f, VND_META_SLOT_MODEL_0);
  Vision.begin(&NN_Instance_nia_v4, nia_pp, cfg);
}

void loop() {
  Vision.run();
}

#Build

On the first build, the #pragma neuron6 directive causes a prebuild hook to locate nia_1_int8.onnx, run ST Edge AI Core on it against the board's memory pool description, and compile the generated network into the binary. The build output prints a summary: the number of layers placed on the NPU, the number executed in software, and the network's memory requirement. This step adds one to two minutes to the first build.

The result is cached per model and configuration. Subsequent builds with an unchanged model skip the step. Changing the model file, its name or its address triggers regeneration.

ST Edge AI Core must be installed; see Installation.

#Upload

Model weights are stored in external flash separately from the application. The uploader writes them before the application, after requesting a checksum of the existing blob from the bootloader and skipping the write when it matches. A rebuild that changes only code uploads in seconds. Tools > Model weights controls this behaviour.

#Operation

PostProcess_NIA_OnSlot(conf, iou, slot) creates the post-processor: the code that converts the NPU's output tensors into a list of boxes with classes and scores, publishes them to Neuro Studio, and draws them on the display when one is present. conf is the confidence threshold below which detections are discarded. iou is an overlap threshold that this model ignores, since it removes duplicates itself. slot is the metadata slot the results are published to.

Vision.begin(model, pp, cfg) initialises the camera, the pipe that scales frames to the 256x256 input, the NPU, the JPEG encoder and the USB stream, and the post-processor.

Vision.run() performs one pass: capture a frame into the model's input buffer, run inference, run the post-processor, publish. Video streams in the background at the camera frame rate; the boxes update at the inference rate.

#The model

nia_v4 is Ohm Lab's object detector, trained in house and licensed with the core. It recognises the 80 COCO classes. Its input is 256x256 RGB. It deduplicates detections with a local maximum test rather than non-maximum suppression, which is why the overlap argument is unused.

Class names come from the shared classes_table. A sketch may redefine the table to rename classes.

#Reading detections

The post-processor writes its results to the global nia_pp_output after each Vision.run():

void loop() {
  Vision.run();

  bool person = false;
  for (uint32_t i = 0; i < nia_pp_output.nb_detect; i++) {
    if (nia_pp_output.pOutBuff[i].class_index == 0) {   // 0 is "person" in COCO
      person = true;
    }
  }
  digitalWrite(LEDB, person ? LOW : HIGH);
}

Each detection has a centre, a width and a height in normalised coordinates from 0 to 1, a class index and a confidence. Bundled Models lists the output structure of each post-processor.

#Related examples

The same folder contains segmentation, pose estimation, hand landmarks and a two-model sketch with the same structure. Changing the model requires changing the pragma, the declaration macro and the post-processor factory.

#Common faults