Cascades and Tracking

A cascade is a two-stage inference in which a detector finds a subject and a second model runs on a hardware crop of that subject at the second model's own input size. A hand 200 pixels wide in a 1024x768 frame occupies about 40 pixels after scaling into a 192x192 detector input; cropped, it fills the second model's input. MediaPipe's hand, face and pose pipelines are built this way.

The core provides three model-independent components.

Component	Function
`Vision.runRoi(roi, model, pp, mode)`	Runs a model on a sub-region: hardware crop and scale into the model's own input
`n6_roi_t` and its builders	Convert a detection into a region (rotate, square, expand) and map results back
`n6_tracker_t`	The detect-then-track state machine: stage selection, smoothing, lock loss

All three are provided by Arduino.h.

runRoi

cpp

bool Vision.runRoi(const n6_roi_t &roi, NN_Instance_TypeDef *model,
                   IPostProcessor *pp, OVL_Mode_t mode = OVL_RELOAD_CLEAR);

The camera pipe is reprogrammed to crop the region in hardware at full sensor resolution and scale it into the model's input. Inference and the post-processor then run as in runWith().

The model's input dimensions are read from the network, so no size constants are required for the second stage.
The base crop is restored on the next run() or runWith().
The crop globals are republished, so the second model's results map to full-frame coordinates through the standard helpers. Second-stage overlays and metadata need no knowledge of the region.

The function returns false and runs nothing if the region is empty or off frame, or the model has no usable input shape.

Regions

cpp

typedef struct {
  float cx, cy;    // centre, camera-normalised
  float w,  h;     // size, camera-normalised
  float angle;     // clockwise radians, 0 upright
} n6_roi_t;

A region is expressed in camera-normalised coordinates, as required by the crop hardware. The builders accept NN-normalised detections and convert using the crop live at the time of the call.

The geometry of a particular model pair is expressed as data:

cpp

typedef struct {
  int8_t rot_kp_start, rot_kp_end;   // keypoints whose vector defines "up"
  int8_t rot_kp_end2, rot_kp_end3;   // optional extra keypoints averaged into the end. -1 if unused
  float  target_angle_deg;           // angle the vector is rotated to
  float  scale_x, scale_y;           // growth, applied after squaring
  float  shift_x, shift_y;           // shift in units of the box size, along its axes
  bool   square_long;                // square to the long side
} n6_roi_spec_t;

Averaging several keypoints into the rotation endpoint reduces jitter in the tracked crop. Keypoint index 0 is valid, so unused slots must be set to -1 explicitly; a designated initialiser that omits them zero-fills and averages in keypoint 0.

cpp

// From a detector box and optional keypoints. kps is an array of x, y pairs
// with a stride: 2 for {x,y}, 3 for {x,y,z}, NULL/0/0 for none.
n6_roi_t n6_roi_build(float cx, float cy, float w, float h,
                      const float *kps, int kp_stride, int kp_count,
                      const n6_roi_spec_t *spec);

// From a landmark set alone. Used by a landmark model to re-aim itself.
n6_roi_t n6_roi_from_nn_keypoints(const float *kps, int kp_stride, int kp_count,
                                  const n6_roi_spec_t *spec);

// A point from the second model back to camera coordinates, undoing rotation.
void n6_roi_unmap_point(const n6_roi_t *roi, float rx, float ry,
                        float *out_cam_x, float *out_cam_y);

The transform order is shift, square, scale, matching MediaPipe and ST's reference implementation. Squaring before shifting computes the shift against the wrong side length.

Tracker

The tracker converts the cascade into a state machine so that the detector does not run on every frame.

cpp

n6_tracker_t tr;
n6_tracker_init(&tr, /*smoothing*/ 0.5f, /*max_misses*/ 5);

bool            n6_tracker_is_tracking(const n6_tracker_t *t);
const n6_roi_t *n6_tracker_roi        (const n6_tracker_t *t);
void            n6_tracker_update     (n6_tracker_t *t, const n6_roi_t *roi);
bool            n6_tracker_miss       (n6_tracker_t *t);   // false once the lock is lost
void            n6_tracker_reset      (n6_tracker_t *t);

It holds three things: the current stage (searching or tracking), an exponential moving average on the region's centre and size with shortest-path interpolation on the angle, and a miss counter that drops the lock after max_misses consecutive empty frames. The first update after acquisition is taken without smoothing. A zero-area region is ignored. The tracker is pure state and several may be kept.

Hand landmarks example

cpp

#pragma neuron6 model="033_palm_detection_full_quant_pc_uf_od.tflite"    name=palm_det
#pragma neuron6 model="033_hand_landmark_full_quant_pc_uf_handl.tflite"  name=hand_lm

static const n6_roi_spec_t PALM_TO_HAND = {        // acquisition
  .rot_kp_start = 0, .rot_kp_end = 2, .rot_kp_end2 = -1, .rot_kp_end3 = -1,
  .target_angle_deg = 90.0f,
  .scale_x = 2.6f, .scale_y = 2.6f,
  .shift_x = 0.0f, .shift_y = -0.5f,
  .square_long = true,
};

static const n6_roi_spec_t HAND_TO_HAND = {        // tracking
  .rot_kp_start = 0, .rot_kp_end = 4, .rot_kp_end2 = 6, .rot_kp_end3 = 8,
  .target_angle_deg = 90.0f,
  .scale_x = 2.0f, .scale_y = 2.0f,
  .shift_x = 0.0f, .shift_y = -0.1f,
  .square_long = true,
};

OV5640 camera;
NEURON6_DECLARE_MODEL(palm_det);
NEURON6_DECLARE_MODEL(hand_lm);
static n6_tracker_t hand;

void setup() {
  VisionConfig cfg;
  cfg.camera    = CAMERA_1080P;
  cfg.fps       = FPS_30;
  cfg.nn.width  = 192;                 // stage one input; stage two reads its own
  cfg.nn.height = 192;
  cfg.nn.aspect = ASPECT_CROP_CENTER;
  cfg.af        = AF_CONTINUOUS;
  Vision.begin(cfg);
  n6_tracker_init(&hand, 0.5f, 5);
}

void loop() {
  if (!n6_tracker_is_tracking(&hand)) {
    Vision.runWith(&NN_Instance_palm_det,
                   PostProcess_PalmDetector_OnSlot(0.5f, 0.3f, VND_META_SLOT_MODEL_0),
                   OVL_CLEAR);
    if (pd_ui_nb_detections > 0) {
      const pd_ui_detection_t *d = &pd_ui_detections[0];
      n6_roi_t roi = n6_roi_build(d->x_center, d->y_center, d->width, d->height,
                                  &d->kps[0].x, 2, AI_PD_PP_NB_KEYPOINTS, &PALM_TO_HAND);
      n6_tracker_update(&hand, &roi);
      vnd_meta_invalidate_slot(VND_META_SLOT_MODEL_0);
    }
  }

  if (n6_tracker_is_tracking(&hand)) {
    const bool ran = Vision.runRoi(*n6_tracker_roi(&hand), &NN_Instance_hand_lm,
                                   PostProcess_HandLandmarks_OnSlot(0.5f, VND_META_SLOT_MODEL_1),
                                   OVL_RELOAD_CLEAR);
    if (ran && handlm_output.valid) {
      n6_roi_t next = n6_roi_from_nn_keypoints(&handlm_output.kps[0].x, 3,
                                               AI_HANDLM_NB_LANDMARKS, &HAND_TO_HAND);
      n6_tracker_update(&hand, &next);
    } else if (!n6_tracker_miss(&hand)) {
      vnd_meta_invalidate_slot(VND_META_SLOT_MODEL_1);
    }
  }
}

The two slot invalidations prevent the idle stage's last result from remaining on the video: a palm box for the duration of tracking, or a skeleton after the hand has left.

The specifications are MediaPipe's published constants. Acquisition rotates the box so the wrist to middle finger vector points up, squares it, scales it by 2.6 and shifts it half a box length towards the fingers. Tracking re-aims from the average of three finger joints with a tighter 2.0 scale.

Other model pairs

Only the specification changes. A face pipeline levels the eyes:

cpp

static const n6_roi_spec_t FACE_TO_LANDMARKS = {
  .rot_kp_start = 0, .rot_kp_end = 1, .rot_kp_end2 = -1, .rot_kp_end3 = -1,
  .target_angle_deg = 0.0f,
  .scale_x = 1.5f, .scale_y = 1.5f,
  .shift_x = 0.0f, .shift_y = 0.0f,
  .square_long = true,
};

A box feeding a classifier, with no keypoints:

cpp

static const n6_roi_spec_t BOX_TO_CLASSIFIER = {
  .rot_kp_start = -1, .rot_kp_end = -1, .rot_kp_end2 = -1, .rot_kp_end3 = -1,
  .target_angle_deg = 0.0f,
  .scale_x = 1.25f, .scale_y = 1.25f,
  .shift_x = 0.0f,  .shift_y = 0.0f,
  .square_long = true,
};
n6_roi_t roi = n6_roi_build(cx, cy, w, h, NULL, 0, 0, &BOX_TO_CLASSIFIER);

A specification that requests rotation with too few keypoints falls back to an upright box. The core ships no presets; a specification belongs with the sketch or library that owns the model pair.

Rotation limitation

The region's angle is used to aim it, so the shift runs along the subject's axis, but the pixels are not rotated. The crop hardware is axis aligned, and runRoi() feeds the region's upright extent. Second-stage accuracy is best with the subject approximately upright, since landmark models are trained on rotation-corrected crops.

The crop is not inflated to the rotated bounding box. Because the pixels are not rotated, enclosing the tilted rectangle would shrink the subject in the input by up to 1.4x at 45 degrees. The specifications carry enough margin for a rotated subject to remain within the upright window.

Pixel rotation would require either the GPU2D, whose drawing library is a closed binary not available to the core, or a CPU warp. Either could be added behind runRoi() without changing sketches, which is why the angle is carried through the API.

Tracking without a model

A bounding box is a region, so n6_tracker_update_box(&tr, cx, cy, w, h) steadies a colour blob, an AprilTag or any jittery detection. See Colour Blobs and Motion.

#runRoi

#Regions

#Tracker

#Hand landmarks example

#Other model pairs

#Rotation limitation