bledev internals: writing a backend
This page is for you if you’re adding a backend (bledev.bleak,
bledev.webble, bledev.cpble) or changing one. What an app can rely
on is in The rules every backend keeps, below, and in the
Bluetooth LE guide; this is how a backend delivers it.
The layout
lib/bledev/ follows audiodev and displaydev: a portable base, one
standalone module per backend, and an optional selector.
Module |
What it is |
|---|---|
|
The contract: |
|
In-process loopback. The reference implementation of every hook. |
|
MicroPython, over aioble. |
|
CPython on Windows, Linux and macOS, over bleak. Central only. |
|
CircuitPython, over |
|
Web Bluetooth in a browser, central only. |
|
Nordic UART byte stream, built only on the contract. |
|
The REPL over nus: MicroPython’s raw |
|
Improv Wi-Fi setup, both sides, built only on the contract. |
|
BLE-MIDI as a usbif-style MIDI port, both sides, built only on the contract. |
|
The BLE-MIDI packet codec. Pure, no imports, shared by every host. |
|
CircuitPython’s BLE file-transfer protocol: |
|
SIG services as classes over |
|
More SIG services, one class each. The cycling and running ones answer their control points through |
|
HID report descriptors and reports to |
|
HID over GATT, host and device, built on the contract and |
|
Picks a backend. Nothing imports it, and backends must not. |
A backend module imports bledev and whatever its host provides, and nothing
else in bledev except the base classes. auto.BACKENDS expects
bledev.bleak.BleakBLE and bledev.webble.WebBLE; if you name your class
differently, change that table.
Anything under lib/ must also be listed in omnipython-desktop.toml (a test
checks it). A central-only host module belongs in mip-split.toml’s
[bledev] host-only list once its file exists, so it doesn’t ship to boards.
The hooks
Subclass these and fill in the hooks. Everything else in the base classes is built on them.
BLE, the adapter:
backend(class attribute): the module’s short name.capabilities(): start fromBLE.capabilities(self)and setcentral,peripheral,mtu,max_mtu,indicate,pairing,l2cap.config(*names, **settings): at leastmtu; raiseUnsupportedErrorfor anything else you don’t have.addr_mode(ADDR_MODE_*) only where the stack has private addresses: mpble passes it to NimBLE and saysUnsupportedErrorwhere the stack refuses it (BTstack).close().Peripheral backends:
register_services(*services), which sets eachCharacteristic._impl(see below), andadvertise(interval_us, *, name, services, appearance, manufacturer, connectable, timeout_ms), which returns yourConnectionwithrole="peripheral". A central-only backend leaves both alone; the base raisesUnsupportedError.Central backends:
scan(duration_ms, *, active, interval_us, window_us), returning aScanner, and_connect(device, timeout_ms, **options), returning yourConnectionwithrole="central".find()is already written on top ofscan().
Scanner: either override __aenter__/__anext__/cancel() (mpble
wraps aioble’s scanner that way), or keep the base’s queue and call
_push(ScanResult(...)) and _finish() from the event loop, starting and
stopping your scan in _start()/_stop(). bleak’s detection callback runs on
the event loop, so _push from it is fine. From another thread it isn’t: the
base uses asyncio.Event.
Device: construct it with your adapter, the address (six bytes, or the
host’s string), and native= your own object, which _connect() gets back.
Connection: is_connected(), disconnect(timeout_ms),
disconnected(timeout_ms), exchange_mtu(mtu, timeout_ms) (return the
current MTU if the host negotiates on its own), the mtu property (override
it, or keep self._mtu current), and _discover_services(uuid, timeout_ms),
returning a list of your ClientService in handle order.
ClientService: _discover_characteristics(uuid, timeout_ms), a list of
your ClientCharacteristic in handle order.
Pairing and descriptors (for HID): Connection.encrypted and
Connection.pair(bond, timeout_ms); ClientCharacteristic._discover_descriptors(uuid, timeout_ms)
returning your ClientDescriptors, whose read() you fill in; and on the
server, each Characteristic’s descriptors (fixed values) and encrypted
flag. Report long_read in capabilities(): False if a read returns only
one packet. A backend without pairing leaves the base’s UnsupportedError.
ClientCharacteristic: read, write, subscribe, notified,
indicated. self._want_response(response) resolves response=None and
checks the flags for you; self._check(FLAG_..., what) checks the rest.
The server side: Characteristic is backend-neutral. Your
register_services() gives each one an _impl object with read(),
write(data, send_update), notify(connection, data) (synchronous),
indicate(connection, data, timeout_ms) and written(timeout_ms). The public
methods check the flags before they call it.
What a backend must translate
Every timeout to
BLETimeoutError.bledev.wait_ms(awaitable, timeout_ms)does it for you.A dead link to
DisconnectedError, including an operation started after the drop. aioble raisedTypeErrorthere; bleak raises its own errors.A full transmit buffer to
BusyErrorfromnotify(), or a wait-and-retry inside an async send.A payload over
mtu - 3toBLEError, checked before it reaches the host stack, because NimBLE and some OS stacks truncate silently.A queue overflow to
BLEErrorfrom the next read of that queue, never a silent drop.Discovery order to handle order.
How mpble gets there
aioble is a good base with three places it loses data, and mpble works around each rather than forking it:
Notifications. aioble keeps one notification per client characteristic and replaces it when the next arrives. mpble swaps in a
_Queueofqueue_limitentries. It also doesn’t use aioble’snotified(), which waits only when the queue holds one item or fewer and assumes the flag is then set: true for a one-slot queue, but with a longer one it strands the last item of a burst. That hung the first board-to-board stream at its tail.Captured writes. aioble shares one ten-entry deque across every characteristic and drops the oldest. mpble registers its own IRQ handler beside aioble’s, copies each captured write out of the value buffer in the IRQ, and ties it to its connection while the connection still exists.
Scans. aioble’s defaults (1.28 s interval, 11.25 ms window) listen less than 1 % of the time. On the bench a six-second scan saw one advertiser, while a full-duty scan saw the board forty times, so mpble defaults to 30 ms interval and window.
Two more turned up on hardware: discovery results come back in IRQ-race
order, and a BufferedCharacteristic loses its initial value, because aioble
writes it and then sizes the buffer. mpble sorts the first and rewrites the
second. From reading aioble rather than from a failure: its advertise()
returns None when cancelled, and mpble turns that back into
CancelledError.
aioble owns the one bluetooth.BLE() and its IRQ, so mpble.get() shares
one adapter, and the capture handler is registered once per boot.
The laptop found two more:
Notifications before a subscription. NimBLE’s server notifies whether or not the central wrote the CCCD, and mpble registered its queue at discovery, so a board central received what a laptop never would. mpble’s queues ignore everything until
subscribe()opens them. The over-the-air check “nothing arrives before subscribe()” failed on the old code.An MTU exchange before the connection. Windows exchanges the MTU the moment it connects, and MicroPython 1.29 on the S3 delivers
_IRQ_MTU_EXCHANGEDbefore_IRQ_CENTRAL_CONNECT(seen every time, instrumented in the IRQ). aioble has no connection to give it to and drops it, so the board believed the MTU was 23 while the link ran at 247, and every notification to a Windows central carried 20 bytes. mpble’s IRQ keeps the early value and applies it on the connect. Laptop downstream went from 19 to 49 KB/s. Upstream’s own order is worth an issue; the board-to-board case never showed it because the central there asks for the MTU after connecting.
The bless probe found two more on the P4, with the board as the central and Windows as the peripheral:
The same early MTU, the other way round. Windows exchanges the MTU the moment the board connects, before aioble has recorded the connection, so the board believed 23 on a 256 link. mpble’s IRQ keeps that value too, and the connection’s
mtupicks it up.exchange_mtu()after the peer already exchanged raisedEALREADY. NimBLE exchanges once per link, so mpble returns the settled MTU.bledev.hid’s host exchanges before reading the Report Map, so any peripheral that exchanges first (a real keyboard may) would have failed.
Code that uses the raw API. A board config’s ble is an MPBLE,
and MPBLE.__getattr__ hands every name it doesn’t define to the raw
bluetooth.BLE, so ble.active() or ble.gap_advertise() still work.
irq(handler) is the exception: it runs the handler from mpble’s IRQ rather
than replacing aioble’s. Raw advertising trips aioble in two places, both
handled in mpble. aioble’s peripheral IRQ calls _connect_event.set() on every
incoming connection and raised AttributeError when aioble.advertise() had
never run, which stopped the IRQ reaching anyone after it; mpble gives it a
flag at startup. And aioble records that connection but never forgets it,
because the task that would is started by its own advertise(); mpble forgets
it on the disconnect. Checked on the T-Embed: raw service, raw advertising,
a user IRQ that saw the connect and disconnect, a laptop read, then nus on
the same adapter.
How bleak gets there
bleak hands each notification to a callback on the event loop. The backend
queues it per characteristic (queue_limit, an overflow raises) in
_BleakClientCharacteristic._received, and routes it to indicated() only
when you subscribed with indicate=True alone, because bleak can’t tell the
two apart. On Windows that subscription passes force_indicate.
The OS negotiates the MTU itself. exchange_mtu() waits up to a second for
the value to move off 23 and returns it; against gatt_peripheral.py, which
asks for 185, Windows settles on 185. BlueZ always reports 23 through bleak,
so the backend also takes the largest write-without-response size it saw in
discovery, plus three.
connect(priority="throughput"), or an interval of 15 ms or less (what
nus.connect() passes on a board), calls WinRT’s
RequestPreferredConnectionParameters(ThroughputOptimized) through bleak’s
private _backend._requester, as the P4 measurements did. On the S3 it made
no difference to nus throughput.
After a disconnect bleak sets client.services to None, so every client call
checks the connection first and raises DisconnectedError.
Reconnecting fast
The REPL gate once failed one run in five: bleak couldn’t subscribe on a
connection made 1.5 s after the last one closed. A loop of rapid reconnects
from the laptop (tests/bledev_board/reconnect_loop.py: log in, run a line,
close, wait 1.5 s) caught it at about one cycle in fifty, always the same
way. bleak’s connect returned in 70-440 ms where a real one takes 1.7 s or
more, Windows reported the device connected and the GATT session active, the
board never saw a connection (its IRQ log shows it advertising throughout),
and the first GATT operation hung until Windows reported a drop about nine
seconds later.
It went away with the REPL’s output fixes (below): against bledev-host’s
repl.py the loop failed 7 times in 465 cycles; against the fixed one, 0 in
270, 150 of them with the retry switched off. We haven’t found how the
board’s old timer handling produced a connection Windows believed in and the
board never saw, so bledev.connect_and_set_up() stays as the defence: it
bounds the setup at 4 s and tries again, up to three times. nus.connect()
and midi.connect() use it, and the contract checks both the retry and the
giving up.
How cpble gets there
_bleio has no events and no async calls. An adapter hands out objects you
look at: adapter.connections, connection.connected, and the ring buffers
that a characteristic’s writes and notifications land in. So every wait in
cpble is a poll of one of those every POLL_MS (5 ms), and each blocking
_bleio call (connect, discovery, a read, a write with response, making a
subscription) holds the event loop until the radio answers, which is one or
two connection intervals. A scan runs in slices of SCAN_SLICE_MS (100 ms),
each a fresh start_scan(), with the scan response merged per address the
way aioble does.
Where writes land. A characteristic with notify or indicate gets a
_bleio.PacketBuffer, which keeps each write whole but takes writes only from
the central that subscribed, and only after it did (it learns the connection
from the subscribe event). Any other writable characteristic gets a
_bleio.CharacteristicBuffer, a byte stream: with capture=True, each
written() returns what arrived since the last one. Both drop the oldest data
when full and don’t say so; cpble checks the stream buffer’s fill and raises
when it’s full, and sizes it at 32 KB (stream_buffer), which is nus’s whole
16 KB echo phase in flight twice over. _bleio doesn’t say which central
wrote, so written() names the most recent live one. Every server value
buffer is _bleio’s largest (512 bytes), because a notification can’t be
longer than the value’s buffer; max_len is kept in Python (read() gives
the first max_len bytes) rather than by _bleio refusing the write.
Notifications, and a stall in CircuitPython 10.3. PacketBuffer.write()
on the ESP32 keeps a packet it couldn’t get an mbuf for and retries it on the
next BLE_GAP_EVENT_NOTIFY_TX. NimBLE sends that event when a notification is
handed to the stack, not when it leaves the radio, so a packet that never got
an mbuf never produces one, and the next write that doesn’t fit beside it
waits inside _bleio until the central disconnects. Writing 253-byte packets
back to back from the laptop gate delivered 10 (2,530 bytes) and stopped,
every time. So cpble passes each notification as the header= of an empty
write: _bleio copies a header only into an empty packet, and a write of
nothing never waits, but it does retry what’s pending. A return of 0 means
the previous packet is still there, and notify() raises BusyError; nus,
midi and the other async senders wait and try again. A task retries the last
packet every 5 ms for 5 s after it was sent, since nothing else will. It
also keeps notifications whole, where PacketBuffer would append the next
write to a pending packet.
And a hard fault. When NimBLE refuses a notification after the mbuf was
allocated (it needs one more for the ATT header), PacketBuffer “undoes” by
switching to a second outgoing buffer that it allocates only for indications
and writes, and the next write copies into a null pointer: the board restarts
in safe mode. cpble can make it rarer but not impossible: notifications are
held to 244 bytes (the reported MTU is at most 247, one link-layer packet, so
NimBLE never fragments), and each new connection gets fresh packet buffers.
The fix belongs in CircuitPython, in the Espressif port’s
_bleio/PacketBuffer.c.
Discovery mode, without a hand. CircuitPython’s own file service
advertises to a new host only in discovery mode, which normally takes a reset
pressed during the blue blink after boot. The supervisor marks that second in
an RTC register that survives a reset, and memorymap can write that register
on the S3 (RTC_CNTL_STORE0_REG, 0x60008050), so
tests/bledev_board/cpfiles_discovery.py writes the mark and resets. It also
erases the board’s bonds, as the button would.
What _bleio can’t do here, on the ESP32 port: a central can’t receive
indications (they’re ignored below Python); pairing is “just works” only
(authenticated=True raises); there are no long reads; the MTU is fixed at
256 when CircuitPython is built; a service has at most 10 characteristics and
a characteristic at most 2 descriptors; a peripheral doesn’t learn its
central’s address; Service.deinit() is the only way to take a service away,
which register_services() uses; and a read that times out inside _bleio
returns a full buffer as if it succeeded. A refused read or write comes back as
BluetoothError("Unknown system firmware error: 261"), NimBLE’s 0x100 plus
the ATT code, and cpble turns that into GattError(5).
How the REPL gets there
The REPL has to work when nothing runs asyncio, at the >>> prompt, so the
board side doesn’t use bledev’s adapter at all. It registers the Nordic UART
service with the raw API (the RX buffer in append mode, 1 KB), advertises
with gap_advertise, and does everything else from an IRQ handler it adds to
aioble’s dispatcher (or sets itself, with no aioble). os.dupterm reads a
_Stream whose readinto returns None when empty, because 0 means end of
stream and detaches it. Output goes out as notifications of mtu - 3 bytes,
and what the controller can’t take yet is retried every 5 ms by one periodic
machine.Timer. A program printing faster than the link carries waits in
write() up to two seconds, then drops the excess, because print() can’t
fail.
Two ways that retry went wrong, both found by printing a 16,890-character list over the link (the REPL gate does this):
A doubled chunk and a lost one. The timer’s callback runs between the main program’s bytecodes, so it could land inside a
flush()thatprint()had started: both sent the same chunk, then both trimmed it, and the next chunk never went out. Five prints out of five came back corrupt.flush()returns at once if it’s already running.A panic. The retry used to be a one-shot timer re-armed on every write. On the esp32,
Timer.init()on a virtual timer whose alarm is being dispatched clears its handler first, soesp_timer’s task on the other core calls a NULL function (InstrFetchProhibitedintimer_process_alarm). It took the board down twice in about 45 long prints. The timer is started once and never re-armed; 60 long prints afterwards, no panic and every byte intact.
Ctrl-C works because the IRQ calls os.dupterm_notify() whenever a write
contains 0x03; dupterm then reads it and raises KeyboardInterrupt, even in
a loop that never reads stdin. It only does that for 0x03: notifying on every
write would push pasted input through the 260-byte stdin ring, which drops what
doesn’t fit.
The password stage is repl.Login, pure and checked on both interpreters.
Nothing reaches dupterm until the password line matches, and anything sent
after a wrong one is dropped. The prompt goes out at connect, but a client
that subscribes after connecting misses it, so an empty line asks for it again
without using up the attempt.
The REPL takes a free dupterm slot on login (the highest, so slot 0 stays
WebREPL’s where there are several) and frees it on disconnect. The ESP32 has
one slot, and WebREPL holds it while it has a client, so a BLE login then gets
The REPL is busy and a disconnect, the way WebREPL turns away a second
client. Both servers run side by side; they only take turns at the REPL.
How file transfer gets there
The protocol is Adafruit’s (supervisor/shared/bluetooth/file_transfer.c in
CircuitPython; the spec is in Adafruit_CircuitPython_BLE_File_Transfer):
service 0xFEBB, a version characteristic reading 4, and one transfer
characteristic the client writes commands to and the board answers with
notifications. bledev adds one characteristic, ADAF0300-..., for the
password; CircuitPython has none, which is how a client tells the two apart.
FileServer is the protocol without a radio. It takes the client’s bytes in
any split, handles every command whose bytes have all arrived (several may
share a write), and queues answers. Its packet(n) never lets one
notification carry the end of one answer and the start of the next, because
Adafruit’s client expects a header at the start of a notification. A read’s
data is streamed from the file one packet at a time rather than loaded.
On the board it’s a second service on repl’s raw-API server, registered
in the same gatts_register_services call as NUS, because a registration
replaces everything before it. What differs from CircuitPython, and why:
File work runs on the main thread, not in the IRQ. On the esp32 port the BLE IRQ runs in NimBLE’s host task (
MICROPY_PY_BLUETOOTH_USE_SYNC_EVENTS), whose stack is a few KB. The first build handled commands there, and a recursive delete took the T-Embed down. The IRQ only appends the bytes to an inbox andmicropython.schedule()s the work; the REPL’s 5 ms timer picks up anything the schedule queue refused. A disconnect in the IRQ only flags the reset, so the main thread never has a file closed under it.The window is 4 KB, not a 512-byte sector. Each
WRITE_PACINGoffersmin(remaining, window)bytes, so 20 KB is five round trips. The transfer characteristic’s append buffer is2 * window + 64, room for a window, its header and the next command, and the board never offers more than that.Truncation. MicroPython’s files can’t truncate, so a write at an offset that leaves the file shorter copies the part that stays.
Locked means answered. Before the password, every command gets its own answer shape with status
0x80, so a client can parse the refusal rather than time out.
Adafruit’s own client (adafruit_ble_file_transfer.FileTransferClient, with
_bleio stubbed and a stand-in PacketBuffer that hands it one packet() per
read) wrote, read, listed, moved and deleted against FileServer at MTU 247,
and wrote and read at MTU 23; its listdir can’t run at 23 against any
server, because it reads a 28-byte header into one 20-byte packet. That was
a one-off check, not a test in the tree. The client against a real
CircuitPython board, whose service pairs, is described under
How cpble gets there.
How HID gets there
bledev.hid is two halves over one parser. bledev.hidreport reads a report
descriptor into Reports of Fields (bit offset, size, count, flags,
usages, logical range) and a Decoder diffs successive input reports into
events. It imports events and keys and nothing from Bluetooth.
The keyboard decoder is usbif’s, event for event. Same order (modifier
releases, key releases, key presses, modifier presses, the new modifier mask
on all of them), same fields (scancode is the HID usage, window is
None), same rollover rule. tests/test_bledev_hid.py feeds 3,000 random
boot reports through both decoders when usbif’s checkout is beside this one,
and they must agree. It found one difference on its first run: the ISO “# ~”
key (usage 0x32), which SDL calls # and usbif leaves unmapped. There is
one table, keys.hid_keycode, and both decoders use it: the key’s code comes
from the same place whether it arrived over USB or Bluetooth.
Mice are the mouse and pointer collections. Relative X and Y add up to a
pointer position that starts at (0, 0) and stays inside bounds if you give
the decoder one; an absolute pointer’s logical range spans the bounds.
Buttons are renumbered to SDL’s (HID’s 2 is right, SDL’s 3). The wheel
(desktop 0x38) is the event’s y and AC Pan (consumer 0x238) its x, which
is what the desktop’s MOUSEWHEEL carries. usbif has no mouse decoder to
match: it hands a hosted mouse’s reports to Python raw.
Axes, hats and buttons come only from joystick, gamepad and multi-axis collections. Axes are numbered by usage (desktop X to wheel, then the simulation page), so an Xbox controller’s sticks are 0 to 3 and its triggers 4 and 5. Every axis starts at 0.0 and moves only when its value changes, as SDL’s do.
Descriptors, encryption and pairing were added to the contract for HID.
A Report characteristic says which report it carries in its Report Reference
descriptor (0x2908), and HOGP gives the report ID nowhere else, so the host
needs ClientCharacteristic.descriptor(). A Characteristic can be
encrypted=True, and Connection.pair() encrypts a link. The host pairs
only after a read fails with insufficient encryption or authentication,
which is how a real keyboard asks. mpble pairs through bledev.security
(“just works”, io=3, unless a passkey is given); ble.enable_bonding()
loads the bond store (NVS on an ESP32, see
how pairing gets there) and turns bonding on, and
it has to run on both sides,
because the side that didn’t start pairing stores keys too.
MicroPython has no long read. gattc_read() is one ATT Read, so a board
central gets at most mtu - 1 bytes of any value. The host exchanges the MTU
up to 247 before reading the Report Map (246 bytes fit), and a map that fails
to parse at that length raises an error that says why. Our own map is 162
bytes; an Xbox Series controller’s is 283, so a board can’t host one until
MicroPython grows a long read. bleak reads long values itself, and the
fake’s long_reads=False models the board.
The peripheral’s identity comes from its functions. Name, appearance and PnP product ID are derived from which of keyboard, consumer control and gamepad it serves, so a host that cached one set never sees another set under the same identity. The PnP vendor is the Bluetooth SIG’s test company ID (0xFFFF), because we have no vendor ID of our own.
A laptop can’t see a HID service. Windows hides 0x1812 from apps: bleak
lists the board’s GAP, GATT, Battery and Device Information services and a
22-handle gap where the HID service sits (measured against the P4). Chrome’s Web Bluetooth blocklist hides it too. So
Peripheral(service_uuid=hid.INSPECT_SERVICE) serves the same
characteristics under a vendor UUID, and Host(service_uuid=...) reads them
there; that is how the laptop checks a board’s HID side, with the same code
a board host runs. No host treats a board serving that UUID as a keyboard.
The host writes the keyboard’s LED report once it has subscribed, as Windows
does. Peripheral.leds() returns those writes, which is how the gate knows
the host is listening, and how it times a round trip.
How pairing gets there
bledev.security is the board side. aioble pairs, but it leaves three gaps,
and all three bit on the LCD-7:
The passkey step is unhandled. aioble’s
_IRQ_PASSKEY_ACTIONhandler only logs. bledev answers it from the IRQ itself, as NimBLE’s own examples do: a random six-digit passkey forDISPLAY(the board shows it), the caller’s passkey forINPUT(a board as central), and numeric comparison throughconfirm(). Only the drawing waits for the main thread.Saving keys through the scheduler fails. aioble stores a secret in the IRQ and saves it with
micropython.schedule(), which raises when the queue is full. The queue is full whenever a scheduled callback runs long while a timer keeps queueing: drawing the passkey (77 ms on the LCD-7) inside a scheduled callback while the REPL server’s 5 ms timer ticks did it. The exception went back into NimBLE as a failed key store, and pairing stopped after the first key, with the host believing it had paired. bledev handles_IRQ_GET_SECRETand_IRQ_SET_SECRETahead of aioble and writes the store inline when the queue is full (ESP-IDF’s own NimBLE writes NVS from its host task too). Anything else deferred from an IRQ goes throughsecurity.poll()when the queue is full, which the REPL server calls every tick.The store is a file. bledev keeps the keys in NVS on an ESP32 (
NVSStore, namespacebledev, one blob: type, key length, key, value length, value, repeated) and in aioble’sble_secrets.jsonelsewhere. It takes over aioble’sload_secretsand_save_secrets, which aioble looks up at call time. The store should be loaded before the radio starts, because NimBLE reads the board’s IRK then;repl.start()does that. When the radio is already on, the board makes a new IRK, which costs nothing while it advertises its public address.
Why NVS. A filesystem reformat, a mip reinstall and “delete everything” are routine on these boards, and a board that silently forgets its bonds leaves every paired host with keys that no longer work. NVS survives all of those and is lost only to a full chip erase, which is rarer and already means setting the board up again. Not measured: an actual filesystem reformat with a bond in NVS; what was shown is that the bond lives outside the filesystem (no file is written) and survives hard resets.
The server. With pairing=, the REPL’s RX and the file service’s
transfer and AUTH characteristics get _ENC flags (just works) or _ENC
plus _AUTHN flags (passkey, numeric), so NimBLE refuses an unpaired host
before bledev sees the request, answering ATT 0x05. bledev checks the link’s
state from _IRQ_ENCRYPTION_UPDATE as well. AUTH is served even without a
password, so the table depends only on console and files, and reads 1
once pairing has unlocked it. With the password gone, the REPL opens on the
client’s first line (_Open), so the client sees one >>>. A link not
paired within PAIR_WITHIN_MS (30 s) is hung up on, so a host that never
pairs can’t hold the board’s one connection. The 30 s start again when the
stack asks about a passkey (security.passkey_asked()): Android asks “Pair?”
before it shows the passkey box, and a person who reads that question for
twenty seconds would otherwise have ten left to type six digits. A failed
pairing isn’t hung up on at once: the host would only see a dropped link, not
why.
The laptop. bleak 3.0.2’s WinRT pair() accepts only CONFIRM_ONLY,
so bledev.bleak runs WinRT’s custom pairing itself: CONFIRM_ONLY and
PROVIDE_PIN, the PIN supplied behind a deferral so the passkey provider can
be slow (a person, or the gate reading the board’s console). No system dialog
appears. WinRT reports protection_level_used as NONE even after a passkey
ceremony, so bledev takes “a PIN was asked for and given” as the evidence of
an authenticated link; the board’s own state agrees (authenticated, 16-byte
key). A device Windows already has paired is left alone: Windows encrypts with
the stored keys when a protected characteristic is first used.
A lost bond, seen from the host. When the board has forgotten the keys,
Windows’ encryption fails and the link drops two milliseconds later (the
board’s log: GET_SECRET misses, ENCRYPTION_UPDATE with encrypted 0,
disconnect). After a board resets, Windows also hands out links the board
never sees, for a few seconds. From the host the two look the same, so
connect_and_set_up backs off between attempts and, when every attempt
drops after encrypting with stored keys, says both.
webble
webble reaches the page the way audiodev.web_audio does, through the js
module, which Pyodide and MicroPython’s WebAssembly build both have. The
runtimes disagree on how Python values cross into JavaScript, so everything
goes through a few helpers made once with js.Function.new(): strings,
numbers and opaque objects cross, and bytes cross as hex. JS errors come back
as each runtime’s JsException, and _translate() maps them by name.
What it takes to meet the contract in a browser, all found on hardware:
Chrome on Android never matches a
namefilter against a scan response. A board advertising nus’s 128-bit UUID has its name pushed there, sofind(name=, service=)filters on the service alone and checks the pick’s name afterwards. The check caught two wrong picks while the S3s were also advertising nus.The MTU is invisible. Chrome on Android asks for 517 right after connecting (logcat:
configureMTU() mtu: 517, answered with 247 by a board), and Windows negotiates on its own, but neither tells the page. Writes are sized toWebBLE(mtu=), default 23. A write longer than the link carries is cut short on Android without an error, which is what the mock models andtest_bledev_web.pyplants.First connects fail. Android gives status 133 / HCI 0x3E (“failed to be established”) on about half the first attempts to the P4, so
_connect()tries three times. One gate run needed two retries.One GATT operation at a time. Chrome rejects a second while one is in flight, so each connection serialises them behind a lock.
Discovery is slow the first time. The browser walks the whole GATT table on the first request, which took longer than nus’s 2 s from Windows, so discovery timeouts have a 10 s floor.
MicroPython’s asyncio can’t
wait_for()a JS promise, only a coroutine, so promises with a timeout are wrapped in one.
The gate is tests/bledev_web/: gate.py runs nus’s three 16 KB phases
against nus_gate_server.py on a board, on three pages (pyodide.html,
mpy.html, wasm.html) served by serve.py, which also collects what the
pages log. ?mock=1 swaps in mock_bluetooth.js, and
tests/test_bledev_web.py runs that in headless Chromium with two planted
faults. Against a board, the chooser is the only human step, and both ends
of it can be automated:
Android:
adb reverse tcp:8765 tcp:8765makes the pagelocalhoston the phone,adb shell input tappresses the page’s button (a real user gesture), anduiautomator dumpfinds the device’s row and the Pair button in the chooser. Chrome needs the Nearby devices permission, and Android’sDeviceAccessevents never fire.Desktop Chrome: start it with
--remote-debugging-portand its own--user-data-dir, click the button withRuntime.evaluate(userGesture=true), and answerDeviceAccess.deviceRequestPromptedwithDeviceAccess.selectPrompt. No person is needed at all.
Restart the board’s server with a hard reset: after a Ctrl-C, a new
nus.serve() on the P4 twice said it was advertising while nothing was on
the air.
The checks
Every behaviour above has a check, and each check has a planted fault that
must make it fail. Over the fake, on both interpreters:
tests/bledev_contract.py
is the contract, run by tests/test_bledev.py
with five planted faults (a dropped, corrupted, reordered, truncated or
overwritten packet). A host backend that has a fake peripheral to talk to can
reuse most of it by building its adapters there instead; if you change the
fake, keep the plants failing. The pairing and file-transfer protocols have
their own scripts and plants beside it.
Over a real radio, the scripts in
tests/bledev_board/
pair a serving script on one board with a client on a second board, a laptop
or a browser. Its README lists each pair, what it checks, and how to run it.
The rules every backend keeps
These hold on every backend. The fake enforces the strict version of each, so code that passes its tests keeps them:
One packet is
connection.mtu - 3bytes. A notification or write longer than that raisesBLEError. Real NimBLE would cut the tail off and say nothing.exchange_mtu()asks for more, and the smaller side wins.A characteristic holds
max_lenbytes, 20 by default, as on MicroPython. Raise it for anything that takes MTU-sized writes. The fake raises on a longer write; a real board keeps the firstmax_lenbytes and says nothing (measured: 29 bytes sent, 20 kept).Nothing is dropped quietly. Notifications and captured writes queue in order. If a queue overflows because you stopped reading, the next
notified()orwritten()raisesBLEErrorinstead.notify()can sayBusyError. It’s synchronous, like aioble’s, and the transmit buffers are finite. Wait a moment and send again, or use nus, which does. Every async send waits for room itself.Subscribe before you expect notifications. Browsers and bleak deliver nothing until you do, and the fake matches them.
Discovery comes back in the server’s order (by handle).
A dropped link ends pending client calls with
DisconnectedError, and anything that arrived before the drop is still delivered first. A server’swritten()serves every connection, so it doesn’t end on a disconnect; give it atimeout_msor cancel it. Withcapture=True,written(timeout_ms=0)returns a write that’s already queued, or raisesBLETimeoutErrorat once.Every timeout is
BLETimeoutError, and every error is aBLEError:UnsupportedError,DisconnectedError,BLETimeoutError,BusyError,GattError(with.status).UUIDis one value everywhere.UUID(0x180F),UUID("180f")and the 128-bit string are equal.str()is always the 128-bit form, which bleak and browsers use, andto_bytes()is the little-endian wire form.
Details the guide leaves out
Behaviour an app meets now and then, which the Bluetooth LE guides leave short.
The REPL over Bluetooth
Any terminal that speaks Nordic UART works too: send an empty line, and the
board answers Password: . A wrong password gets Access denied and a
disconnect, and nothing sent with it reaches the REPL. The link isn’t
encrypted, as with WebREPL, so someone nearby with a sniffer can read the
password, unless you pair.
It runs beside WebREPL. On a board with one console slot (the ESP32), the
two take turns: while a WebREPL client is connected, a Bluetooth login gets
The REPL is busy and a disconnect, the way WebREPL turns away a second
client, and the other way round.
While it runs, the REPL owns the radio: it registers its own service and
advertises whenever nobody is connected. Call bledev.repl.stop() to give the
radio back to your app.
Files over Bluetooth
It’s locked the way the REPL is. CircuitPython uses pairing; bledev uses the REPL’s password by default (or pairing), sent to one extra characteristic, and until it’s right every command is refused. One attempt per connection, and a wrong one is hung up on. Logging in to the REPL over the same connection unlocks files too, which is what mcuctl does. The protocol has no checksum beyond the radio’s own, so check what matters (mcuctl compares SHA-256 after every transfer).
start() serves the REPL beside the files; console=False serves files
alone. It’s MicroPython only, and like the REPL it owns the radio while it
runs.
How fast: 20 KB from a laptop to an ESP32-S3 in about 2 s and back in about
1.5 s, or about 1.2 s and 0.5 s when the laptop connects with
priority="throughput".
MIDI ports
The port is the contract the midiports package
documents: read(nbytes) returns MIDI bytes or None and never blocks,
readinto(buf, nbytes) does the same into your buffer without allocating,
any() says how many bytes are waiting, and write(buf, nbytes) sends.
Every message comes out whole, with its status byte, and a read takes only
the whole messages that fit. Give adafruit_midi an in_buf_size bigger than
your longest SysEx: its 1.6.0 parser throws away a SysEx that reaches it in
two reads.
If you’d rather have messages than bytes, with the sender’s timestamp, use
ts, message = await port.receive() or async for ts, message in port.
Messages you write go out from a task, packed as many to a packet as fit, and
await port.drain() waits for them.
On a board, ask for the shortest connection interval when you connect:
midi.connect(ble, name="synth", min_conn_interval_us=7500, max_conn_interval_us=7500). The latency that buys is in
How fast it goes.
HID host and bonding
A keyboard that wants an encrypted link gets one: the first read it refuses
pairs the link (“just works”, no passkey) and tries again. On a board, call
ble.enable_bonding() once before connecting so the keys are kept, and the
next connect doesn’t pair from scratch. The keys live in NVS on an ESP32
(see where the keys live), so a full chip erase
forgets them, and a keyboard that remembers the board may then need putting
back into pairing mode.
HID device details
Install adafruit_hid on MicroPython with
mip.install("github:PyDevices/omnipython/bledev_hid"): it fetches
Adafruit’s files at a release tag, with one file patched so
consumer_control imports on MicroPython, and micropython-lib’s
__future__. Each send_report() returns once the radio has the report,
waiting briefly if its buffers are full. The keyboard’s LED report (Caps
Lock and friends) reaches kb.led_on(Keyboard.LED_CAPS_LOCK) the way it does
over USB.
adafruit_hid has had no gamepad since 5.0, so bledev.hid_gamepad gives
one in the shape of Adafruit’s hid_gamepad.py example, for a
Peripheral(ble, gamepad=True): press_buttons(), move_joysticks(),
set_hat(), and set() for everything in one report.
Its name, appearance and Report Map come from which functions you ask for,
the same way every time, because hosts cache a device against its identity.
Change the set, and a host that paired with the board before must forget it
and pair again. Pass encrypted=True to make hosts pair first, as real
keyboards do.
HID presence key and the parser
A paired phone as a presence key. A phone reconnects to a bonded HID device by itself whenever its Bluetooth comes back in range, so a board can tell when you walk up, with no app on the phone. Pair once from the phone’s Bluetooth settings, then wait for each arrival:
ble = bledev.auto.adapter()
ble.enable_bonding()
per = hid.Peripheral(ble, keyboard=False, mouse=False, encrypted=True, name="Presence")
while True:
conn = await per.serve() # the phone is here
screen_on()
await conn.disconnected() # the phone has gone
screen_off()
Android gives the phone a new random address over time, so watching
advertisements for a known address doesn’t work; the bonded reconnect does.
conn.encrypted is still False at the moment serve() returns. A bonded
phone re-encrypts the link about 0.3 s later, so wait for it before you treat
the visitor as the phone you paired.
Don’t pair a test board with the computer you’re testing from. Once paired, Windows and phones use it as a real keyboard, and it types into them.
The parser underneath, bledev.hidreport, knows nothing about Bluetooth.
Give it any HID report descriptor and it decodes that device’s reports, so a
USB host can use it the same way. How it numbers axes, what it does with a
rollover, and what a board central can’t read are in
bledev-internals.md.
A changing address, watching it
A scanner sees the new address each time the old one expires, so keep scans going for a quarter of an hour to watch it change. Closing the adapter puts the mode back to the stack’s default.
Pairing, in detail
The first time a computer connects, the board draws a six-digit passkey on its
screen (and prints it to the console) and the computer asks for it. The board
asks for pairing as soon as a host connects, so a phone puts up its pairing
dialog straight away; that’s how Workbench in Chrome on Android pairs. After that
the two are bonded: they reconnect encrypted, with no passkey, until one of
them forgets. password=False means the passkey is the only lock; keep a
password as well if you like.
From a laptop or another board, pair when you connect, and pass the passkey, or a function that asks for it:
link = await bledev.repl.connect(ble, name="rack", pair=True, passkey=lambda: input("Passkey: "))
files = await bledev.filetransfer.connect(ble, name="rack", passkey=ask) # pairs when the board asks it to
On Windows that’s WinRT’s own pairing, with no system dialog, and it pairs
before the link comes up. Once a link is up, Windows answers the board’s
request to pair on its own, out of sight, and pairing then fails without asking
for the passkey, so on Windows pass pair=True rather than waiting for the
board to ask. To pair a
computer once for tools that don’t pair themselves (mcuctl),
python -m bledev.bleak pair rack asks for the passkey in the terminal, and
python -m bledev.bleak unpair rack undoes it. Windows Settings, Add device,
works too.
Just works on its own would let anyone in range pair and get a REPL, so it
needs the password too; what it adds is that nobody sniffing can read the
password or the session. A board draws the passkey on board_config.display_drv;
pass show(passkey) and hide() to put it somewhere else.
A board’s name, and the services it serves, follow from what you start: the
name you pass, or the board’s name when you pass none, is both
the advertised name and the one a paired computer lists
(the GAP name, which would otherwise be MicroPython’s “MPY ESP32”), and the
same console and files give the same table every time, whatever the lock,
because hosts cache a paired device’s table. Serving something else on the same
board (HID, say) changes it under the same address, and MicroPython doesn’t tell
bonded hosts, so a host that kept the old table may need the board unpaired and
paired again.
Standard services
Register your own service in the same call: register_services() replaces what
came before. The HID peripheral serves
Battery and Device Information itself. These are plain GATT tables, so any
backend that can be a peripheral serves them. examples/bledev_battery_simpletest.py
shows both running.
The cycling and running services each have a control point that a central
writes and the board answers by indication. Run sensor.run() as a task to
answer it. MicroPython waits for the central’s acknowledgement; CircuitPython
sends the indication but _bleio doesn’t report the acknowledgement, so
indicate() returns once it’s queued. Each module’s docstring lists what the
measurement carries and which flags it sets, and each has an example under
examples/ named bledev_<service>.py that a phone fitness or sensor app can
read.
File transfer speed and offsets
write() takes an offset, and the file ends after what you wrote.
read() takes one too. The board paces every write with a free-space count,
so it never receives more than it can hold.
How fast: 20 KB from a laptop to an ESP32-S3 in about 2 s and back in about
1.5 s, or about 1.2 s and 0.5 s when the laptop connects with
priority="throughput".
BLE-MIDI packets
The packet format lives in bledev.midi_codec, pure Python with no
imports: timestamps, running status, SysEx split across packets, and clock
slipped into a SysEx. Its test vectors (tests/bledev_midi_vectors.json) were
worked out by hand from the specification, and every interpreter checks
against the same file.