How esp-vision’s modules are built
Use esp-vision’s modules says how to
build with tflite and vision. This page says what is behind them.
The pin and the glue
modules/.esp-vision
fetches esp-vision at its full commit in DEPENDENCIES.lock. Its fetch.sh links a
sibling clone at that commit (../esp-vision), or fetches that commit alone.
The leading dot keeps it out of --modules and all, because esp-vision’s own
micropython.cmake is its whole firmware build, not a module.
modules/tflite and modules/vision are our glue around esp-vision’s files:
the ESP-IDF components esp-vision uses, at its versions; the include paths and
definitions MicroPython’s QSTR scan needs (it sees only a module’s own paths,
never a linked component’s); and five pieces of our own.
imlib compiles inside the module, not as an ESP-IDF component. As a component it would wait for MicroPython’s generated headers while being one of their inputs (a ninja dependency cycle). esp-vision’s own
components/imlibCMakeLists can’t be used anyway: it finds its per-board headers byMICROPY_BOARD, which names upstream’s generic boards here. Ours takes them from one of esp-vision’s boards per chip (ESP32_P4X_FUNCTION_EV_BOARD,ESP32_S3_EYE).src/tflite_streaming.cppletstfliterun streaming models. It includes esp-vision’spy_tflite.cppunchanged, with two TensorFlow Lite Micro types swapped for subclasses: an interpreter that owns 20 resource variables (their bookkeeping in a 1 KB arena on a 32-bit chip, as ESPHome’smicro_wake_wordsets up), and an op resolver that registersCALL_ONCE,VAR_HANDLE,READ_VARIABLEandASSIGN_VARIABLE. Every other operator those models use is in esp-vision’s list already.src/vision_preview_none.cstands in for esp-vision’s preview, which streams frames to its IDE over its own USB multiplexer.esp-vision-patches/holds two small changes to esp-vision’spy_espdl.cpp:espdlreleases the GIL while a model runs, and its module table listsHumanFaceDetectfromsrc/vision_espdl_face.cpp. The build applies them, in order, to a copy of the file in the build tree, so the pinned checkout stays as it is, and a pin move that breaks one stops the build with the patch’s name.src/vision_h264.cish264’s encoder, in place of esp-vision’splatform/h264.c. It is that file with one addition: the encoder in a P4 before revision 3 reads only packed YUV420 and refuses RGB565, so firmware built for those chips converts each frame in software first. Revision 3 and later take the image’s RGB565 as esp-vision does.
Two build details follow from esp-vision’s code. The prebuilt RTSP library calls
ESP_ERROR_CHECK’s handler, which nothing else references with MicroPython’s
assertions off, so vision links it with -u _esp_error_check_failed, the way
the port does for abort_. And ESP-DL’s headers are C++20, so the module’s C++
compiles with -std=gnu++2b, esp-vision’s own setting.
What was checked on hardware
Test |
P4 DEV-KIT |
S3, octal PSRAM |
|---|---|---|
|
pass, worst error 0.016 |
pass, outputs identical |
|
104 µs |
397 µs |
|
pass; JPEG 42 ms |
pass; JPEG 84 ms |
|
pass, 68 ms, IoU ≥ 0.90 |
pass, 176 ms, IoU ≥ 0.92 |
|
Constrained Baseline, frames as drawn |
(P4 only) |
|
151 frames in 10 s at 15 fps |
(P4 only) |
Each test fails when it should: the hello-world test with its inputs offset, the pedestrian test with one person blacked out.
Streaming models were checked on an ESP32-P4 board (Waveshare’s ESP32-P4 WiFi6 Touch LCD 4B). ESPHome’s “Hey Jarvis” model was fed the int8 feature slices of six clips: three with the wake word, two ordinary sentences and loud noise. Its 1467 outputs were identical, byte for byte, to a TensorFlow Lite Micro reference running the same model on the same slices on a PC. One call, three slices, takes 1.5 to 2 ms there. The comparison sees a change of one step: with the reference keeping one clip’s model state into the next, 11 outputs differed, each by 1 of 255.
What it costs
Built into an all image, the two modules added about 2.5 MB on the S3 (the
LCD-7’s app partition grew from 0x470000 to 0x6d0000, leaving about 1 MB of
filesystem on 8 MB of flash) and about 3.1 MB on the P4 (firmware.bin 4.6 MiB
to 7.6 MiB of 16 MB). Most of it is ESP-DL. That is why they are opt-in.
Not included
sensor(the camera). esp-vision’s camera code is per board, and every P4 board’scamera.cuses Espressif’s Board Manager (its YAML board definitions). The other route: esp-vision’s genericplatform/camera.cdeclares the camera functions weak, so our glue could provide them from cameraif, which already drives the OV5647.display. esp-vision’s display code also needs Board Manager, and displayif already does displays here.One
esp_h264. h264enc pins 1.4.1 andvisiontakes the same; esp-vision uses 1.3.0.sensorwould bringesp_video, which requires 1.3.*, so a firmware with h264enc and a camera viaesp_videodoesn’t resolve.