How casting works
This is what happens under the castdisplay guide: the Miracast session, the encoder on the board, how sound joins the picture, and the limits that come from them.
The Miracast session
Miracast here is Wi-Fi Display over infrastructure (Microsoft’s MS-MICE): the
board and the screen are on the same Wi-Fi network, not on a direct link.
castdisplay.micecast runs the session in Python with one poll loop and no
threads: MICE on TCP 7250, then the Wi-Fi Display RTSP exchange on 7236, then
RTP over UDP from the server port it advertised (Windows opens its firewall
for that port only). A Roku needs MICE’s Session Request first, and answers
an early capability query with an error, so the session waits for the sink’s
own OPTIONS before asking.
Once the sink says PLAY, the stream is a C task on the P4’s core 0, the
firmware’s castif. It converts the framebuffer with the PPA, encodes H.264
in hardware with h264enc,
muxes MPEG-TS with tsmux
and sends it over RTP, with the audio as 48 kHz LPCM on the same clock.
Python only answers keepalives, forwards the sink’s keyframe requests and
draws. castdisplay.castfast is the Python side of that task.
On CPython, castdisplay.pycast stands in for castif: ffmpeg encodes the
H.264, and Python muxes the same MPEG-TS and sends the same RTP, 7 TS packets
to a datagram, byte for byte what the P4 sends. Windows’ receiver checks the
stream strictly, so matching a stream it’s known to accept matters. A test
compares pycast’s muxer with castif’s output.
Input from a laptop
Windows sends the laptop’s mouse, keyboard, touchscreen and remote over
Miracast’s back channel (UIBC) as HID reports. castdisplay.uibcinput reads
each device’s report descriptor, integrates the mouse’s relative moves into a
cursor, turns clicks and fingers into touches and keys into keys codes, and
merges them into the board’s own touch_read and keypad_read.
Sound with the picture
On the P4, sound reaches the cast in one of two ways. castfast.PumpFeed
makes 10 ms blocks for the speaker and the cast at once, paced on the clock.
Or castif reads the audio pump’s output itself (castfast.TapFeed), so
whatever an app plays is what the sink gets, and nothing the interpreter does
can starve it. CastDisplay(audio=True) uses the tap. The pump drives the
I2S wire in stereo even when the app opened a mono stream, which is why
audio_format defaults to (24000, 2).
A 10-minute drum machine cast measured about 1 ms of A/V drift, with the audio clock within 13 ppm of the PC’s and nothing dropped on the board.
HLS
HlsDisplay on the P4 hands each finished frame to castif.Hls, a task on
core 0 that encodes, cuts one-second segments and serves the playlist and
segments over HTTP without the interpreter. It takes a frame only once it has
started on the last one, so a fast app costs one copy per encoded frame. On
CPython, ffmpeg encodes and segments and a thread serves. Each run serves
under its own path, because a TV caches by URL.
Limits
No flash writes while casting. A littlefs write erases a block about every two minutes, which parks core 0 and stalls the cast. Log to the console while a cast runs.
A whole-panel refresh flashes the P4’s glass white. Draw, then refresh just the rows you changed (
display_drv.flush_rect).Weak Wi-Fi is audible. At -70 dBm about 0.4 % of packets were lost; at -80 dBm several percent. Put the board near the access point.
The P4’s H.264 encoder keeps its reference rows in one block of internal RAM, 1,152 bytes for each 16 pixels of width, so a long-running board can have the memory free but in pieces. Cast early, or restart first.
Measuring a cast
tools/rtp_capture.py
records a cast on a PC, and
tools/ts_timing.py reports
what a sink would hear: lost packets, audio gaps, the audio and picture clocks
against the PC’s, and the A/V offset second by second. Run the capture with
Windows Python when the PC is WSL, because a WSL socket doesn’t see the LAN’s
UDP:
python.exe tools/rtp_capture.py capture.bin 5004 700
python tools/ts_timing.py capture.bin