I built a tiny line-crossing people counter for a Seeed XIAO ESP32S3 Sense this week. Camera on the board, model on the board, counts on the board. No laptop in the loop unless I want the more accurate host-mode version.
The first useful constraint was admitting what the hardware could actually do.
The host detector runs at 416×416 and is about 1.1 GFLOPs. That is fine on a laptop. On an ESP32-S3 it is fantasy. The on-device detector is roughly 15× smaller and still takes about 118 ms per frame.
At QVGA the budget ended up like this:
JPEG decode ~30 ms
preprocess 9 ms
model 118 ms
postprocess 2 ms
---------------------
total ~160 ms
So the real target is not “real time” in the desktop-CV sense. It is about 6 fps, which is enough for someone walking through a doorway if the rest of the pipeline does not get clever.
The counter is deliberately plain:
- capture the newest frame
- detect people
- use the bottom-center of each box as the foot point
- match points with a nearest-centroid tracker
- count when a track crosses a configured line
- log the event to serial, optionally expose
/countsover WiFi
The foot point mattered more than I expected. Box centroids move around with posture and crop shape. If the counting line is on the floor, the bottom-center of the box is a better approximation of the thing that actually crosses the line.
The tracker is also intentionally boring:
// ponytail: no motion model, no re-id. Fine for a doorway at ~6 fps;
// swap in ByteTrack if people cross paths in frame.
class Tracker {
// id -> centroid, max distance, max age
};
No Kalman filter. No re-identification. No cloud fallback. Just keep a track alive for a few missed frames and drop it when it gets too stale.
That is not because those techniques are bad. It is because this is a doorway counter on a microcontroller. People mostly move in one direction, the camera points down, and the useful failure mode is obvious: if people cross paths in frame, this tracker is too small and should be replaced.
That shape is easier to operate than a “smart” pipeline pretending it has more signal than it does.
The board can serve its own tiny monitoring page when WiFi is configured:
GET /counts
{"in":14,"out":11,"occupancy":3}
But it also works serial-only with empty WiFi settings. That was another good constraint. The core counting path should not depend on a dashboard, a backend, or a phone app being alive.
The bit I like most is that the counting logic has a host self-check separate from the ESP-IDF build:
make -C firmware/test
# ok: 6 counter tests
For edge AI, that split is worth doing. The model and camera are the annoying parts to test. The line-crossing rules do not need to be. Pull the state machine into boring C++, beat it up on the host, then let the board worry about frames and inference.
The lesson was the same one on-device AI keeps teaching me: the model is only one budget. Latency, memory, camera format, mounting angle, testability, and failure honesty matter just as much.
A smaller model plus a boring counter beats an impressive pipeline that cannot keep up with someone walking through a door.
