Software/Arduino Core/Camera

Streaming the Camera

Streaming live video from the camera to Neuro Studio without a neural network, and the settings that apply.

beginnerOV5640VD66GY3 min read

A camera-only sketch declares the sensor and calls Vision.begin() with a configuration and no model. The camera image is encoded to JPEG and sent to the host by background tasks. The examples on this page declare the OV5640; a VD66GY sketch includes VD66GY_Arduino.h and declares VD66GY camera; instead, and the remainder is unchanged except where noted.

cpp
#include <OV5640_Arduino.h>

OV5640 camera;

void setup() {
  VisionConfig cfg;
  Vision.begin(cfg);       // camera and USB video pipeline, no model
}

void loop() {
  delay(1000);             // frames are sent by background tasks
}

Vision.run() is not called because it drives inference and no model is bound. A capture task encodes each frame and a USB task transmits it. loop() has no work.

Carrier selection

OV5640 camera; selects the standard Neuro Vision carrier. OV5640 camera(CARRIER_TFT); selects the TFT carrier, which has the 800x480 display and inverted CSI data lanes. An incorrect carrier argument results in no frames rather than a distorted image. A camera initialisation failure is reported on the serial console with the sensor name and expected chip ID.

Settings

The CameraStream example sets every camera field explicitly:

cpp
VisionConfig cfg;
cfg.camera      = CAMERA_XGA;         // 1024x768
cfg.mirror_flip = CAMERA_AUTO_FLIP;   // orientation chosen per carrier
cfg.fps         = FPS_DEFAULT;        // native maximum for the mode
cfg.transport   = TRANSPORT_USB;      // to Neuro Studio
cfg.af          = AF_CONTINUOUS;      // autofocus
cfg.strobe      = STROBE_DISABLED;
Vision.begin(cfg);
FieldEffect
cameraSensor mode and the size of the streamed JPEG. WVGA (800x480) is the default. See Resolutions and Frame Rates
mirror_flipCAMERA_AUTO_FLIP selects the orientation appropriate to the declared carrier. The other values force a flip, a mirror, both or neither
fpsFrame rate. FPS_DEFAULT is the mode's maximum. A request above the maximum is clamped with a console warning
transportDestination of the video. USB requires nothing further. WiFi transports require the WiFi library. TRANSPORT_LCD sends the image to the display and requires the TFT carrier
afOV5640 only. AF_CONTINUOUS refocuses continuously, AF_SINGLE focuses once at start, AF_DISABLED leaves the lens unchanged. The first use loads focus firmware into the sensor and takes about one second. Ignored on the fixed-focus VD66GY
strobeBoth sensors. STROBE_ENABLED pulses the flash LED with each exposure on carriers that have one
isp_enabledVD66GY only. Selects image signal processor tuning; see The Global Shutter Camera

Fine detail such as a QR code or a printed marker requires autofocus to be enabled on the OV5640.

Host display

Neuro Studio shows the stream at the selected resolution with the frame rate and JPEG size. The terminal panel shows the serial console. With Tools > Debug console On, a [RATE] line reports the sensor (cap), encoder (enc) and USB (tx) frame rates once per second. On a healthy stream enc and tx are equal.

Streamed frame rate

The streamed rate is the sensor's frame rate for the selected mode. The sensor, the JPEG encoder and the USB link are pipelined, so an encode overlaps the previous frame's transmission, and the encoder and link keep pace with the sensor at every mode including 5 MP. The STM32N6's VENC encodes JPEG at up to 300 Mpixel/s, which exceeds the pixel rate of any supported sensor mode, and it accepts pixels streamed directly from the camera pipeline without a full frame buffer. The per-mode ceilings for each sensor are listed in Resolutions and Frame Rates.

Host camera controls

With an OV5640 fitted, Neuro Studio presents a Camera Controls panel for exposure, gain and white balance. The same controls are available from the sketch; see Capture Controls. The VD66GY reports no such controls and the panel is not shown.

Access to pixels

Streaming and processing are independent. NN_capture_snapshot() copies one RGB888 frame, at the size and crop set by the nn fields, into a buffer supplied by the sketch. See Snapshots and Image Primitives.