Thermal Camera

The Neuro Vision Thermal module carries a FLIR Lepton 3.1R radiometric thermal camera. The NeuroVisionThermal library implements the Lepton's command and control interface over I2C, its VoSPI video interface over SPI1, and streaming of thermal frames to Neuro Studio as the module's main video.

#Interfaces

The Lepton is held in reset and shutdown until both control lines are driven high. powerOn() does this and is called once in setup(). A full power cycle through powerOn() also recovers a camera that has stopped responding.

#Streaming example

The ThermalCamera example, under File > Examples > NeuroVisionThermal, streams the sensor over USB. On this module there is no colour camera, so the thermal image is the video Neuro Studio displays.

#include <NeuroVisionThermal.h>
#include <PostProcess.h>

NeuroVisionThermal thermal;
static uint16_t tframe[NVT_VOSPI_PIXELS];   // 160x120, 16-bit

void setup() {
  thermal.powerOn();
  thermal.setFlip180(true);                  // the sensor is mounted inverted

  bool ok = thermal.isPresent() && thermal.begin(3000) && thermal.ping();
  thermal.streamBegin();                     // starts the USB video stream
}

void loop() {
  if (thermal.readFrame(tframe, 2000)) {
    thermal.streamFrame(tframe, true);       // true: radiometric, hundredths of a kelvin
  }
}

readFrame() blocks without busy waiting and returns each new frame at about nine frames per second. streamFrame() sends it to the host as a META_THERMAL_FRAME record. With the radiometric flag set, pixel values are temperatures in hundredths of a kelvin; otherwise they are the sensor's display values.

A freshly powered Lepton emits blank frames for several seconds. The example power-cycles the sensor after a long run of failed reads.

#Command interface

The CCI provides:

• isPresent(), begin(timeout), ping(): presence, boot wait and a SYS ping.
• Register access and the GET, SET and RUN command protocol.
• Attributes: serial number, uptime, focal plane array and housing temperature, AGC state.
• Flat field correction (FFC).

The Lepton periodically closes an internal shutter for FFC. Frames within about a second of an FFC are degraded.

#Bus silence during boot

The Lepton fails if it observes I2C traffic to other addresses during its boot window. The bus is claimed for the duration:

n6_i2c_claim();
thermal.powerOn();
delay(2500);
n6_i2c_release();

Once streaming, the Lepton tolerates shared traffic. See I2C with Wire.

#Diagnostics

videoStats() returns counters for the VoSPI path: header packets, segments, frames, discards and resynchronisations. videoLastError(), videoSckHz() and videoSpiErrors() report the link state. The example prints these every few seconds.

Because setup() runs before Neuro Studio connects, the example stores its start-up results in static strings and reprints them from loop().

Two bench sketches, ThermalI2cVerify and ThermalVoSpiTest, exercise the command and video interfaces separately.

#Sketches without Vision.begin

The USB console is normally serviced by the task that Vision.begin() starts. A sketch that never calls Vision.begin() calls streamBegin(), which starts the USB stream without a camera pipeline. A sketch with neither calls TinyUSB's tud_task_ext() from its own loop; the ThermalI2cVerify bench sketch shows the pattern.

#Inference on thermal frames

Running a neural network alongside the thermal camera contends for CPU time with the VoSPI parser, which has a hard deadline. At 45 inferences per second the thermal rate collapsed from about 8.8 to between 2 and 5 fps. cfg.nn.max_fps caps the inference rate; about 22 fps keeps both paths healthy.

#Power

The power save library controls the Lepton's PWR_DWN_L and RESET_L lines through thermalPower(bool). The sensor draws about 150 mW while powered and requires about 950 ms of boot plus 190 ms of VoSPI synchronisation per power cycle. It does not boot across a Stop mode sleep, so duty-cycled sketches power it down between frames. See Low Power.

#Uploading over LTE

The A7683EVideo library's LteThermalUpload example sends a thermal frame, a camera image and a status record to a web server over a cellular modem every thirty seconds. See LTE Modem.