Snapshots and Image Primitives

CPU vision on the Neuro N6 uses NN_capture_snapshot() to obtain a frame and the N6Imgproc library to operate on it. The NPU is not involved.

#Capturing a frame

#define DIM 256

static uint8_t g_snapshot[DIM * DIM * 3]
  __attribute__((aligned(32), section(".ext_psram")));

void setup() {
  VisionConfig cfg;
  cfg.camera    = CAMERA_VGA;
  cfg.nn.width  = DIM;              // snapshot size
  cfg.nn.height = DIM;
  cfg.nn.aspect = ASPECT_CROP_CENTER;
  cfg.af        = AF_CONTINUOUS;
  Vision.begin(cfg);                // no model
}

void loop() {
  if (NN_capture_snapshot(g_snapshot, sizeof(g_snapshot))) {
    // g_snapshot holds one RGB888 frame, DIM by DIM
  }
}

Vision.begin(cfg) without a model still configures the neural network pipe, so the cfg.nn fields determine the size and crop of the snapshot. Cropping and scaling are performed in hardware. Under TRANSPORT_LCD, LCD_GetCameraBuffer() returns the live panel buffer instead.

Two attributes of the buffer are required. It is placed in .ext_psram because a buffer of this size does not fit in on-chip RAM, which the camera needs for its own buffers. It is aligned(32): the DMA rejects destinations not aligned to 16 bytes, and section alignment does not guarantee object alignment, since objects pack at their natural alignment and a neighbouring byte array can displace the buffer. A misaligned buffer is reported once on the console as [NNSNAP].

#Greyscale conversion

static uint8_t g_gray[DIM * DIM] __attribute__((section(".ext_psram")));

n6img_rgb888_to_gray(g_snapshot, g_gray, DIM, DIM);

#Primitives

N6Imgproc operates on caller-owned buffers without allocation or hidden state. The header is n6_imgproc.h.

static uint8_t  g_bin    [DIM * DIM] __attribute__((section(".ext_psram")));
static uint8_t  g_scratch[DIM * DIM] __attribute__((section(".ext_psram")));
static uint16_t g_labels [DIM * DIM] __attribute__((section(".ext_psram")));

n6img_stats_t st;
n6img_statistics(g_gray, DIM, DIM, NULL, &st);   // NULL region = whole image

uint32_t bins[256];
n6img_histogram(g_gray, DIM, DIM, NULL, bins);
const uint8_t t = n6img_otsu(bins);
n6img_threshold(g_gray, g_bin, DIM, DIM, t, 255);

n6img_erode (g_bin, g_scratch, DIM, DIM, 1);
n6img_dilate(g_bin, g_scratch, DIM, DIM, 1);

n6img_component_t comp[8];
const int n = n6img_find_components(g_bin, g_labels, DIM, DIM, 50, comp, 8);
// comp[0] is the largest; boxes are normalised 0 to 1

Wherever an n6img_rect_t * is accepted, NULL denotes the whole image. A rectangle extending beyond the image is clipped.

n6_blob_find_rgb888() clusters a coarse density grid and returns one blob; it is fast and approximate. n6img_find_components() labels pixels and returns every component; it is slower and exact.

#Motion detection

static uint8_t g_prev[DIM * DIM] __attribute__((section(".ext_psram")));

n6img_absdiff(g_prev, g_gray, g_scratch, DIM, DIM);
n6img_statistics(g_scratch, DIM, DIM, NULL, &st);
if (st.mean > 4) { /* the scene changed */ }
memcpy(g_prev, g_gray, sizeof(g_prev));

This test gates expensive work on scene change. The N6Events library packages the same principle as n6_motion_gate_t. See Events and SD Logging and Low Power.

#Performance

Reads from .ext_psram are uncached, and each primitive makes at least one pass over its input. Processing is restricted to the smallest region that answers the question. A 256 or 320 pixel snapshot at a few frames per second is typical; a full 1080p pass is comparatively expensive.

#Self test

ImgprocSelfTest in extras/bench runs every primitive against known inputs without a camera and serves as a reference for the calling conventions.