How esp-vision’s modules are built

Use esp-vision’s modules says how to build with tflite and vision. This page says what is behind them.

The pin and the glue

modules/.esp-vision fetches esp-vision at its full commit in DEPENDENCIES.lock. Its fetch.sh links a sibling clone at that commit (../esp-vision), or fetches that commit alone. The leading dot keeps it out of --modules and all, because esp-vision’s own micropython.cmake is its whole firmware build, not a module.

modules/tflite and modules/vision are our glue around esp-vision’s files: the ESP-IDF components esp-vision uses, at its versions; the include paths and definitions MicroPython’s QSTR scan needs (it sees only a module’s own paths, never a linked component’s); and five pieces of our own.

  • imlib compiles inside the module, not as an ESP-IDF component. As a component it would wait for MicroPython’s generated headers while being one of their inputs (a ninja dependency cycle). esp-vision’s own components/imlib CMakeLists can’t be used anyway: it finds its per-board headers by MICROPY_BOARD, which names upstream’s generic boards here. Ours takes them from one of esp-vision’s boards per chip (ESP32_P4X_FUNCTION_EV_BOARD, ESP32_S3_EYE).

  • src/tflite_streaming.cpp lets tflite run streaming models. It includes esp-vision’s py_tflite.cpp unchanged, with two TensorFlow Lite Micro types swapped for subclasses: an interpreter that owns 20 resource variables (their bookkeeping in a 1 KB arena on a 32-bit chip, as ESPHome’s micro_wake_word sets up), and an op resolver that registers CALL_ONCE, VAR_HANDLE, READ_VARIABLE and ASSIGN_VARIABLE. Every other operator those models use is in esp-vision’s list already.

  • src/vision_preview_none.c stands in for esp-vision’s preview, which streams frames to its IDE over its own USB multiplexer.

  • esp-vision-patches/ holds two small changes to esp-vision’s py_espdl.cpp: espdl releases the GIL while a model runs, and its module table lists HumanFaceDetect from src/vision_espdl_face.cpp. The build applies them, in order, to a copy of the file in the build tree, so the pinned checkout stays as it is, and a pin move that breaks one stops the build with the patch’s name.

  • src/vision_h264.c is h264’s encoder, in place of esp-vision’s platform/h264.c. It is that file with one addition: the encoder in a P4 before revision 3 reads only packed YUV420 and refuses RGB565, so firmware built for those chips converts each frame in software first. Revision 3 and later take the image’s RGB565 as esp-vision does.

Two build details follow from esp-vision’s code. The prebuilt RTSP library calls ESP_ERROR_CHECK’s handler, which nothing else references with MicroPython’s assertions off, so vision links it with -u _esp_error_check_failed, the way the port does for abort_. And ESP-DL’s headers are C++20, so the module’s C++ compiles with -std=gnu++2b, esp-vision’s own setting.

What was checked on hardware

Test

P4 DEV-KIT

S3, octal PSRAM

tflite: TFLM’s own hello-world test (its int8 model, four golden inputs, 0.05 tolerance)

pass, worst error 0.016

pass, outputs identical

tflite: esp-vision’s sine model, per prediction

104 µs

397 µs

vision: examples/vision_smoke.py (draw, read back, find_blobs, JPEG out and in, flush)

pass; JPEG 42 ms

pass; JPEG 84 ms

espdl: ESP-DL’s pedestrian example photo, against ESP-DL’s published boxes

pass, 68 ms, IoU ≥ 0.90

pass, 176 ms, IoU ≥ 0.92

h264: 30 frames decoded on the PC by ffmpeg

Constrained Baseline, frames as drawn

(P4 only)

rtsp: ffmpeg on the PC pulls the stream

151 frames in 10 s at 15 fps

(P4 only)

Each test fails when it should: the hello-world test with its inputs offset, the pedestrian test with one person blacked out.

Streaming models were checked on an ESP32-P4 board (Waveshare’s ESP32-P4 WiFi6 Touch LCD 4B). ESPHome’s “Hey Jarvis” model was fed the int8 feature slices of six clips: three with the wake word, two ordinary sentences and loud noise. Its 1467 outputs were identical, byte for byte, to a TensorFlow Lite Micro reference running the same model on the same slices on a PC. One call, three slices, takes 1.5 to 2 ms there. The comparison sees a change of one step: with the reference keeping one clip’s model state into the next, 11 outputs differed, each by 1 of 255.

What it costs

Built into an all image, the two modules added about 2.5 MB on the S3 (the LCD-7’s app partition grew from 0x470000 to 0x6d0000, leaving about 1 MB of filesystem on 8 MB of flash) and about 3.1 MB on the P4 (firmware.bin 4.6 MiB to 7.6 MiB of 16 MB). Most of it is ESP-DL. That is why they are opt-in.

Not included

  • sensor (the camera). esp-vision’s camera code is per board, and every P4 board’s camera.c uses Espressif’s Board Manager (its YAML board definitions). The other route: esp-vision’s generic platform/camera.c declares the camera functions weak, so our glue could provide them from cameraif, which already drives the OV5647.

  • display. esp-vision’s display code also needs Board Manager, and displayif already does displays here.

  • One esp_h264. h264enc pins 1.4.1 and vision takes the same; esp-vision uses 1.3.0. sensor would bring esp_video, which requires 1.3.*, so a firmware with h264enc and a camera via esp_video doesn’t resolve.