The Global Shutter Camera (VD66GY)

The STCam and Thermal modules carry an ST VD66GY image sensor in place of the OV5640. The VD66GY is a 1.5 megapixel global shutter sensor with 2.61 um pixels. It outputs raw Bayer data over two-lane MIPI CSI-2 at a fixed 804 Mbps per lane. The core presents it through the Vision API; moving a sketch between the two sensors changes the include and the camera declaration.

#include <VD66GY_Arduino.h>
#include <PostProcess.h>
#pragma neuron6 model="yolov8n_256_quant_pc_ii_seg_coco-st.tflite" name=yolov8_iseg

VD66GY camera(VD66GY_CARRIER_THERMAL);   // VD66GY camera; on the STCam module
NEURON6_DECLARE_MODEL(yolov8_iseg);

void setup() {
  VisionConfig cfg;
  cfg.camera      = CAMERA_VD66_FULL;    // full 1.5 MP frame
  cfg.fps         = FPS_45;
  cfg.nn.aspect   = ASPECT_CROP_CENTER;
  cfg.transport   = TRANSPORT_USB;       // no display on these modules
  cfg.af          = AF_DISABLED;         // fixed focus
  cfg.isp_enabled = VD66_isp_tuning_auto;
  Vision.begin(&NN_Instance_yolov8_iseg, PostProcess_YOLOv8_ISEG(0.4f, 0.5f), cfg);
}

void loop() {
  Vision.run();
}

#Global shutter

A rolling shutter exposes the frame line by line, skewing fast-moving subjects. A global shutter exposes every pixel simultaneously. This is relevant for sensors on moving platforms, conveyors and strobe-lit scenes. The strobe output is aligned with the exposure.

#Carriers

VD66GY camera; selects the STCam module, which uses the standard Neuro Vision wiring. VD66GY camera(VD66GY_CARRIER_THERMAL); selects the Thermal module, which has inverted CSI lanes. Neither module has a display; TRANSPORT_LCD warns and produces no output. Neither has the WiFi module; a WiFi transport falls back to USB with a warning. The STCam has a strobe LED and a time of flight sensor; the Thermal module has neither.

#Modes

The sensor is read out as crops of the 1124x1364 array.

The modes are readout bound: the whole array is read each frame regardless of the crop, so VGA is not faster than XGA, and the full frame tops out at 45 fps. FPS_DEFAULT selects the ceiling. The CSI link rate is fixed across modes.

#Image signal processor

The OV5640 delivers processed YUV pixels. The VD66GY delivers raw Bayer data, and the STM32N6 DCMIPP image signal processor performs demosaicing, black level correction, white balance, gamma and colour correction. ST lists the ISP's functions as bad pixel removal, decimation, black level, exposure, demosaicing, colour conversion, contrast, crop, downsize, region of interest, gamma, YUV conversion and pixel packing, operating on three parallel pipes from one input stream. cfg.isp_enabled selects how the ISP is tuned.

The middleware is contained in the VD66GY library and compiles only into sketches that include it. On an OV5640 sketch the field is ignored. The CameraAwbMode bench sketch compares automatic white balance between two modules.

#Differences from the OV5640

• No autofocus. The lens is fixed and cfg.af is ignored.
• No exposure, gain or white balance controls on the camera object. controlCaps() reports none and Neuro Studio's Camera Controls panel is not shown.
• Strobe is supported on the STCam module. Vd66gyStrobeTest in extras/bench verifies it. The Thermal module has no strobe LED.
• CAMERA_1080P and CAMERA_QSXGA map to the full frame.
• The full frame is portrait, 1024 wide by 1344 high. ASPECT_CROP_CENTER provides a square centre crop.

#One-shot capture

The power save library provides PowerSaveCam, which initialises only the sensor and the DCMIPP, captures a single RGB888 frame and returns the sensor to standby, without starting the encoder, the USB stream or the background tasks. It supports the VD66GY only. See Low Power.

#Implementation

The sensor driver and ISP middleware are ST's, vendored under libraries/VD66GY/extras/st-driver. ST's original layout is kept alongside the flattened copy that the build compiles, so that a future update can be compared directly. Two firmware patch blobs are uploaded to the sensor's internal microcontroller at start and cannot be regenerated. The ISP tuning table for the ST module's optics is used; the two other tables ST supplies are for reference optics and would require ST's IQTune tool to adapt.