Running Several Models

Vision.run() executes the model bound at begin(). Vision.runWith(model, pp, mode) executes any model with any post-processor without changing the bound state. Two calls in loop() run two models on the same frame.

#Example

The CascadedModels example runs segmentation and pose estimation on the same view:

#include <OV5640_Arduino.h>
#include <PostProcess.h>
#pragma neuron6 model="yolov8n_256_quant_pc_ii_seg_coco-st.tflite"  name=yolov8_iseg
#pragma neuron6 model="yolov8n_256_quant_pc_uf_pose_coco-st.tflite" name=yolov8_mpe

OV5640 camera;
NEURON6_DECLARE_MODEL(yolov8_iseg);
NEURON6_DECLARE_MODEL(yolov8_mpe);

IPostProcessor *iseg_pp = nullptr;
IPostProcessor *mpe_pp  = nullptr;

void setup() {
  VisionConfig cfg;
  cfg.nn.width = 256;  cfg.nn.height = 256;   // shared by both models

  iseg_pp = PostProcess_YOLOv8_ISEG_OnSlot(0.4f, 0.5f, VND_META_SLOT_MODEL_0);
  mpe_pp  = PostProcess_YOLOv8_MPE_OnSlot (0.4f, 0.5f, VND_META_SLOT_MODEL_1);

  Vision.begin(cfg);                          // no bound model
}

void loop() {
  Vision.runWith(&NN_Instance_yolov8_iseg, iseg_pp, OVL_CLEAR);
  Vision.runWith(&NN_Instance_yolov8_mpe, mpe_pp, OVL_RELOAD);
}

#Metadata slots

Results are sent to Neuro Studio as metadata attached to each video frame. Each post-processor writes to a slot, and the background task composes the current contents of every slot into each frame. Two post-processors on one slot overwrite each other, so each is given its own through the _OnSlot factory variant. Four slots exist: VND_META_SLOT_MODEL_0 to VND_META_SLOT_MODEL_3.

Slot 0 carries the frame-wide box list, the class labels and the inference time display. Other slots carry only their own annotations: masks, keypoints, skeletons. In the example, segmentation occupies slot 0 because its box list covers all classes, and pose adds skeletons on slot 1. The reverse assignment would lose every box that is not a person. This follows from the wire format, which carries one box list per frame with masks and labels referring to entries in it by index.

#Overlay modes

The third argument to runWith() controls the display overlay on that call.

The sequence clear, draw, draw, commit shows both models' output together. The modes have no effect without a display.

#Stale results

runWith() publishes to a slot and marks it ready. The slot remains ready and continues to be sent until it is republished or invalidated. A model that runs only occasionally leaves its last result visible on the video after it has ceased to apply.

Either every model runs on every pass, or a slot is invalidated when its model is about to stop running:

vnd_meta_invalidate_slot(VND_META_SLOT_MODEL_1);

The hand landmarks example invalidates a slot when the detector hands over to the tracker and again when the tracker loses the hand.

#Behaviour of runWith

runWith() is synchronous. It captures one snapshot into the model's input buffer, runs inference while yielding to other tasks on a semaphore, runs the post-processor, draws the overlay according to the mode, and publishes to the slot. Meanwhile the capture task encodes frames at the camera rate and the USB task transmits them, so the visible frame rate does not depend on the number of models.

A post-processor is initialised on its first use, so begin(cfg) without a model followed by runWith() calls requires no separate initialisation. Up to eight distinct post-processors are tracked.

Models executed through runWith() share the pipe and must agree on cfg.nn.width, cfg.nn.height and the aspect mode. A second model with a different input size is run with runRoi(); see Cascades and Tracking.

#Results

Each post-processor writes its output to its global after runWith() returns: pp_output, mpe_pp_output, nia_pp_output, pd_ui_detections, handlm_output. These are read between the calls. See Bundled Models.