Bring Your Own Model

A custom model is added to a sketch by placing the .tflite or .onnx file beside it and adding a pragma and a declaration:

#pragma neuron6 model="my_model.tflite" name=my_net
NEURON6_DECLARE_MODEL(my_net);

The build compiles the model for the NPU, the upload flashes its weights, and Vision.begin(&NN_Instance_my_net, pp, cfg) runs it. Correct output depends on four properties of the model file.

#Tool version

The NPU runtime in the core is ST Edge AI 4.0. A model compiled with any other version links but does not execute correctly. The compilation runs on the host, so the installed version is the one that matters; the build's model summary prints it.

#Quantisation

The NPU executes integer networks. The model is exported with full integer quantisation, int8 or uint8 weights and activations, using a representative calibration set. Float layers fall back to software execution, reported as SW epochs in the model summary. A few SW epochs at the head or tail of a network are normal; a predominantly SW network runs slowly.

#Input contract

Every model compiled through the pragma is given a uint8, channel-last input, because the camera pipe writes packed interleaved RGB888 into the input buffer without CPU conversion. This re-encoding is correct only when the model's input quantisation is asymmetric over the range 0 to 1: scale 1/255 with zero point 0 (uint8) or -128 (int8). A raw byte then dequantises to its value divided by 255.

A model with a symmetric input quantisation, scale 1/127 and zero point 0, is re-encoded to uint8 with zero point 128. A camera byte then dequantises to (b minus 128) divided by 127, so black becomes approximately -1 rather than 0, and the network receives an incorrect image.

The input quantisation is checked before compilation: the input QuantizeLinear scale and zero point in ONNX, or the input tensor's quantisation parameters in TFLite. The bundled MediaPipe models use 1/255 and 0.

The correction is made at export. Rewriting the input quantisation of a calibrated network is arithmetically consistent in isolation but leaves every downstream layer calibrated against the old input grid. Measured on a 416-pixel detector over COCO val2017, a post-hoc rewrite scored 0.297 mAP against 0.328 for the symmetric original and 0.329 for a recalibrated asymmetric export. The model is re-exported with the asymmetric input and recalibrated.

Networks trained on inputs from -1 to 1, such as the bundled MobileFaceNet, are correctly served by a symmetric encoding.

#Input layout

TFLite dimensions follow the NHWC convention. A model converted from an NCHW framework may carry a [1,3,H,W] input, which the compiler interprets as height 3, width W and W channels. It then generates a planar interface while the camera writes interleaved pixels.

The generate report's input line must read [b:1,h:H,w:W,c:3]. A channel count other than 3 indicates the fault. The core's tools/tflite_input_to_hwc.py corrects the common case by moving the graph input past the leading transpose. It was used to produce the bundled face detector.

#Input size

Vision does not read the bound model's input size reliably, so it is set in the configuration:

cfg.nn.width  = 320;
cfg.nn.height = 320;

runRoi() reads the model's size directly, so a second-stage model requires no constants.

#Post-processor

A model with the same head as a bundled one uses that post-processor. Any other model requires a post-processor of its own; see Custom Post-Processors. Raw NPU Output describes inspecting the output tensors while a decoder is developed.

#File location

The pragma searches the sketch directory, then a models/ directory beside the sketch, then the core's models/ directory.

#Flash placement

The external flash weights pool is 26.5 MB, from 0x90300000 to 0x91D80000, shared by all models in a sketch. Models are placed in pragma order at 64 KB alignment. address=0x9XXXXXXX pins a model; the address is compiled into the generated code and forms part of the cache key, so changing it forces regeneration. A sketch may declare up to eight models, each with a distinct name.

#Model summary

Minimal build output prints one line per model: total layers, layers on the NPU, RAM and flash use. Verbose output prints the epoch table with each epoch marked HW, Hybrid or SW. ModelBench in extras/bench measures per-frame inference time on hardware.

#Memory

Activations are placed in on-chip RAM and PSRAM according to the memory pool description in system/npu. A large model alongside a large camera mode and the display can exceed the available memory; Vision.begin() then prints a diagnostic and does not start the camera. Reducing the camera mode or setting cfg.band_loc to PSRAM releases on-chip memory.

#Training considerations

• Training on letterboxed or centre-cropped images and running with ASPECT_CROP_CENTER avoids distortion mismatch.
• A square input avoids aspect handling.
• Input sizes of 256 or below keep inference time and pipe work low.
• Quantisation calibration on images from the same camera, lens and lighting improves accuracy.