Bundled Models

The core's models/ directory contains eight neural network files, and the PostProcess library provides a post-processor for each. Every post-processor factory returns an IPostProcessor* and has an _OnSlot variant that selects the metadata slot for multi-model sketches.

#Summary

conf is the confidence threshold. iou is the intersection-over-union above which two boxes are merged. Typical values are 0.25 to 0.6 and 0.3 to 0.7 respectively. The Nia detector ignores iou.

#Nia

Ohm Lab's COCO-80 object detector. Under USB and WiFi transports it streams detections to Neuro Studio; under TRANSPORT_LCD it draws per-class coloured boxes and labels on the panel. The model was trained on letterboxed images, and ASPECT_CROP_CENTER may improve detections near the frame edges.

The decoder reads six INT8 tensors, dequantises them with the scale and zero point read at run time, applies a sigmoid, decodes boxes, and deduplicates with a 3x3 local maximum test on objectness. No non-maximum suppression runs on the board.

struct {
  uint32_t nb_detect;
  struct { float x_center, y_center, width, height, conf; uint32_t class_index; } *pOutBuff;
} nia_pp_output;

NIA416 is the same family with a 416x416 input. Its model file is not bundled.

#YOLOv8 instance segmentation

Boxes with a per-object pixel mask and class name. Masks are drawn as translucent fills on the display and sent as compressed bitmaps to Neuro Studio. Results are in pp_output with a mask pointer per detection. Mask post-processing on the CPU makes this the heaviest bundled model per frame.

#YOLOv8 pose

Detects people and returns 17 body keypoints each, in COCO order: 0 nose, 1 and 2 eyes, 3 and 4 ears, 5 and 6 shoulders, 7 and 8 elbows, 9 and 10 wrists, 11 and 12 hips, 13 and 14 knees, 15 and 16 ankles. Each keypoint has an x and y from 0 to 1 and a confidence. Neuro Studio draws a skeleton. Because pose depends on body proportions, ASPECT_CROP_CENTER is the appropriate aspect mode.

const int n = mpe_pp_output.nb_detect;
if (n > 0) {
  const mpe_pp_keyPoints_t *nose = &mpe_pp_output.pOutBuff[0].pKeyPoints[0];
  Serial.print(nose->x); Serial.print(","); Serial.println(nose->y);
}

#Palm detector and hand landmarks

Google MediaPipe models in ST's packaging. The palm detector finds hands in a 192x192 input and returns a box and seven keypoints: wrist, four finger bases and two thumb joints. Results are sorted best first.

typedef struct {
  float prob;
  float x_center, y_center, width, height;
  pd_ui_point_t kps[7];
} pd_ui_detection_t;

The hand landmark model runs on a 224x224 crop of one hand and returns 21 keypoints with x, y and relative depth z: 0 wrist, 1 to 4 thumb, 5 to 8 index, 9 to 12 middle, 13 to 16 ring, 17 to 20 little finger, ordered base to tip. It also returns a presence score and a handedness value.

typedef struct {
  bool  valid;
  float score;
  float handedness;      // 0 left, 1 right
  handlm_point_t kps[21];
} handlm_result_t;

The two are designed as a cascade; see Cascades and Tracking. The palm wrapper generates the anchor grid ST's decoder requires and works around two defects in the ST library. The landmark wrapper distinguishes the model's two identically shaped outputs by magnitude and its two scalars by order; a define, AI_HANDLM_SCORE_IS_SECOND, exists for re-exports that reorder them.

#Face detection and recognition

CenterFace detects faces at 128x128 and returns a box and five landmarks. MobileFaceNet converts an aligned 112x112 crop into a 128-value L2-normalised embedding, so a dot product between two embeddings is their cosine similarity. The FaceID library wraps both and provides an in-RAM bank of enrolled faces. See Face Recognition.

#Audio event detection

ST's YAMNet-derived classifier for ten sound classes, fed from the microphone through a log-mel frontend. It has its own API in the N6AudioAI library. See Audio Event Detection.

#Licences

nia_1_int8.onnx is Ohm Lab's and is MIT licensed with the core. The two YOLOv8 files come from ST's model zoo and derive from YOLOv8, whose upstream licence is AGPL-3.0; the terms applying to ST's converted files are being confirmed. The MediaPipe models are Apache 2.0. The face models derive from research networks with permissive licences whose exact source checkpoints are not recorded. The audio model is ST's under SLA0044. The core's models/README.md records the open questions. See Licensing.

#Flash placement

Weights are placed in the external flash weights pool, 26.5 MB from 0x90300000, in pragma order at 64 KB alignment. address= in a pragma pins a model. Changing an address forces regeneration, because the address is compiled into the generated code.