Use esp-vision’s modules
Two modules come from espressif/esp-vision
(Apache-2.0; its imlib is OpenMV’s, MIT), compiled from esp-vision’s own
files. Both are ESP32-P4 and ESP32-S3 only, both need ulab, and both are
opt-in: --modules all leaves them out, so name them:
./build.py --port esp32 --board ESP32_GENERIC_P4 --variant C6_WIFI --flash 16MB --modules ulab,tflite,vision
On a P4 before chip revision 3, build the PRE_REV3_C6_WIFI variant instead.
h264 works there too, but that chip’s encoder reads only YUV420, so each
frame is converted in software first: 30 frames at 320x240 took 635 ms to
encode on a revision 1.3 chip, against 263 ms on a revision 3 one.
tflite: TensorFlow Lite Micro inference (tflite.Model), the API OpenMV’smlmodule uses.vision: the OpenMV-styleimageAPI (image, withimage.ImageIO), ESP-DL inference (espdl), and on the P4 the hardware H.264 encoder (h264) and an RTSP server (rtsp).
esp-vision’s stubs (stubs/*.pyi in esp-vision) document the API. The desktop
ports skip both modules, and they build for MicroPython only.
Built into an all image they add about 2.5 MB on the S3 and 3 MB on the P4,
most of it ESP-DL, so check that the image fits.
image.flush() raises OSError("preview flush failed: ESP_ERR_NOT_SUPPORTED"),
because esp-vision’s preview streams frames to its own IDE and this build has no
such connection.
Streaming models
tflite also runs streaming (stateful) models, such as the microWakeWord
models ESPHome publishes. They keep their state between calls in resource
variables, and each Model has room for 20. Feed one a few slices at a time
and read its output after each predict().
Their input is already int8, so fill the tensor with a callable rather than an
ndarray, which predict() would quantize again. The callable gets the input
tensor’s own memory, its shape and its dtype:
group = bytearray(stride * 40) # the next stride slices, int8
def fill(buffer, shape, dtype):
buffer[:] = group
probability = model.predict([fill])[0][0][0] # uint8: 255 is certain
tflite_streaming_simpletest.py
runs one, and
tflite_wakeword.py
listens for a wake word, with audiodsp’s audiofeatures making the slices
and ESPHome’s detection rules in Python.
tflite_wakeword_mic.py
does the same live from the board’s microphone. It reads the microphone with
board_peripherals.pcm_in() and hands each block to the frontend’s feed():
an audiopump Tap sees only what the board plays, so it can’t carry a
microphone.
Models that run
These ran on an ESP32-P4 board, each with an example. The arena is what
Model.ram reports, and the time is one predict().
Model |
Example |
Arena |
Time |
|---|---|---|---|
microWakeWord hey_jarvis |
19,728 bytes |
1.8 ms |
|
micro_speech (yes, no) |
7,840 bytes |
6 ms |
|
yamnet-cry-distill-int8 (521 sounds) |
128,128 bytes |
28 ms |
|
person_detection |
102,416 bytes |
50 ms |
|
MobileNet-v2 int8 (ImageNet) |
2,608,176 bytes |
1.5 s |
Each example’s docstring says where to download its model.
Detect in a thread
espdl lets go of the interpreter while a model runs, so detection on a
_thread leaves the main thread free: a display, a UI or a recording loop
keeps running through each detect(), classify() or predict(). Only the
image preparation before the model and the results after it hold the
interpreter. Use each model object from one thread at a time, and don’t call
its deinit() while it is detecting.
espdl_thread.py
detects on a thread and measures the main loop’s longest pause meanwhile.
Faces with landmarks
espdl.HumanFaceDetect(msr_path, mnp_path, *, score=0.5, nms=0.5, mnp_score=0.5, mnp_nms=0.5)
runs ESP-DL’s two-stage face detector: MSR finds candidates in the whole
picture, then MNP looks at each one closely and places five key points.
detect(img) takes an RGB565 image.Image and returns a list of
(x, y, w, h, score, keypoints), where keypoints is ten ints, five (x, y)
pairs in ESP-DL’s order: left eye, left mouth corner, nose, right eye, right
mouth corner. set_thresholds() takes the same four keywords as the
constructor, and the whole detection runs without the interpreter’s lock.
The two model files come from ESP-DL’s
models/human_face_detect,
in its models/p4 or models/s3 folder: human_face_detect_msr_s8_v1.espdl
and human_face_detect_mnp_s8_v1.espdl. They are MIT-licensed, by Espressif;
the firmware doesn’t carry them, so copy them to the board’s filesystem and pass
their paths. Release 0.5.0 of the espressif/human_face_detect component is
the one for the ESP-DL this firmware is built with.
espdl_face_landmarks.py
marks the faces in a photo.
Run long scripts as main.py
An mcuctl exec that outlasts mcuctl’s wait resets the board over USB
(rst:0x17 CHIP_USB_UART_RESET). Under a long-running script that looks like
the firmware crashing. Run such scripts as main.py, or with nothing attached.
How the modules are built into the firmware is in How esp-vision’s modules are built.