At the edge, the number that matters is memory - not throughput (specially in Ramageddon)

We rebuilt LF Edge eKuiper in Rust and ran it against eKuiper, Telegraf and Redpanda Connect on five real MQTT workloads — one core, 1 GB of memory, and output checked message-by-message. The most important result wasn't speed.

I-Dacs Labs Engineering · ~16 min read

Most stream-processing benchmarks you'll read optimize for one number: peak throughput on a big server. That number is close to useless for the place these engines actually run — an industrial gateway, an ESPHome hub, a vehicle head-unit, an EV charger. There, you get one or two CPU cores and a few hundred megabytes of free memory, your input arrives over MQTT, and your traffic is bursty in the worst way: fleets reconnect together, chargers start sessions together, devices flush buffered readings all at once after an outage.

In that world two questions decide whether your pipeline survives, and neither is peak throughput:

1. Does the engine keep up on a single core?
2. Does its memory stay bounded when traffic grows?

We built a stream engine called rekuiper to answer "yes" to both, and then we built a benchmark honest enough to tell us if we'd actually managed it. This post is about the benchmark as much as the engine, because the benchmark taught us more than we expected — including a correctness bug in our own code that a throughput-only test would have rewarded as "fast."

The headline: across five MQTT workloads shaped like real deployments, rekuiper produced complete, correct output at 100,000 messages per second on one core in every workload — the top of our tested range, so we never found its ceiling. But the result we care about most isn't that. It's that on the windowed workloads, rekuiper's memory stayed between 5 and 10 MB while the Go-based engines climbed to half a gigabyte to a full gigabyte, or failed. That gap is the whole point, and it comes from design, not from the language.

What rekuiper is

rekuiper is a stream-processing engine written in Rust that reimplements the surface of LF Edge eKuiper: its REST API, its SQL dialect, its stream and rule definitions, and its kuiper command-line interface. The goal was boring on purpose — existing eKuiper rules, the eKuiper Manager web UI, and deployment tooling should keep working — so that "switch the engine" isn't also "rewrite everything."

Concretely, the compatibility surface covers eKuiper's REST API (98 paths and 140 operations, checked black-box against eKuiper's own OpenAPI description), the SQL dialect including JSON paths, CASE, array indexing and unnest, and eKuiper's stream option names (DATASOURCE, FORMAT, CONF_KEY, SCHEMAID, TIMESTAMP, and so on). If you know eKuiper, you already know rekuiper.

What's different is underneath, and it's built around one principle: memory stays bounded under load. Three design choices carry that, and each one shows up later in the numbers.

Three design choices that keep memory flat

Bounded queues with real backpressure

Sources publish records into an in-process stream bus with bounded per-subscriber queues — 4,096 records each. Admission is reserve-then-commit: a batch first reserves capacity in every subscriber's queue, and only then commits. So a batch is either delivered to all subscribers or rejected outright, never half-delivered, and a slow rule pushes back on its source instead of quietly dropping data. Each rule runs as its own task, and its output drains through a bounded sink queue (default 10,000) served by a dedicated sink worker.

The MQTT source uses the rumqttc client, and when a single network read surfaces several publishes, the source admits everything already buffered as one batch of up to 1,024 records. That avoids a per-message wakeup without ever waiting around for more data — you pay one scheduling cost for a burst instead of one per message. This batch-admission trick is a big part of why rekuiper uses roughly half the CPU per message of the Go engines on the simple workloads.

Incremental window aggregation: O(groups), not O(messages)

This is the important one. When you compute GROUP BY device, TUMBLINGWINDOW(ss, 10) with count, avg, max and friends, the naive way is to buffer every row that falls in the window and aggregate at the trigger. Memory then grows with traffic — messages per window — which is exactly the thing that explodes when a fleet reconnects.

rekuiper instead keeps one accumulator per group per aggregate and never stores the rows. Window memory becomes a function of the number of devices, not the number of messages. For the common edge shape — group columns, plain columns, and count/sum/avg/min/max over simple expressions — this incremental evaluator does the whole job. Statements that genuinely need the rows (collect(), joins, some HAVING) fall back to a buffered evaluator, and a unit test checks the two produce identical output on mixed data.

For a fleet, this is the difference between memory that scales with how many vehicles you have and memory that scales with how fast they're all talking at once. Only one of those is safe on a 1 GB box.

An offline sink cache that spills instead of dropping

For intermittent uplinks — a vehicle in a tunnel, a remote site on flaky cellular — a sink can enable a cache using eKuiper's own options (enableCache, memoryCacheThreshold, maxDiskCache, and the rest). Records whose send fails recoverably are queued FIFO: in memory up to a threshold, then in disk pages, and only when the disk budget is exhausted are the oldest records dropped — and counted, not silently lost. The MQTT sink holds one persistent connection per action and reports disconnection, so an outage is detected and cached rather than quietly discarded. (The cache is covered by an integration test but isn't part of the performance numbers here.)

The benchmark that doesn't lie to you

Here's the uncomfortable truth about a lot of edge stream-processing comparisons: they measure throughput at the point the engine acknowledges ingest, or they count output records without checking that the records are correct. Both can hide loss and duplication completely. An engine that drops 15% of your data can look fast if you never verify what came out the other end.

So we built the benchmark around exact output verification, and gave every engine the same cramped room to work in.

Equal, realistic limits. Every engine runs in a container pinned to one CPU core with 1 GB of memory and no swap (--cpuset-cpus=2 --cpus=1 --memory=1g --memory-swap=1g). A separate Mosquitto broker gets its own cores and generous queue limits, so the broker is never the bottleneck. An open-loop Rust load generator (mqttgen, standard library only, MQTT 3.1.1, QoS 0) feeds every engine from the same schedule, and a step only counts if the generator actually stayed on schedule.

Four engines. rekuiper v0.425-beta, eKuiper 2.4.1, Telegraf 1.40.0, and Redpanda Connect 4.109.0 (formerly Benthos). We deliberately excluded Apache Flink: neither Flink 2.x nor Apache Bahir ships an MQTT connector, so testing Flink would have meant a custom source or a Kafka bridge — changing the very ingest path under test. Rather than benchmark a different pipeline and call it Flink, we left it out and said so.

Five workloads shaped like real deployments:

• W1 — telemetry filter. 1,000 devices, one topic, a simple WHERE temp > 21.0 with a unit conversion. Stateless.
• W2 — per-device windows. 1,000 devices, 10-second tumbling windows with count/avg/max. Stateful.
• W3 — ESPHome states. 10,000 plain-text topics via wildcard, using FORMAT="binary" and meta(topic) to carry the topic through. Stateless but wide.
• W4 — vehicle windows. 10,000 topics (one per VIN), 10-second tumbling windows. Stateful and wide — the hardest memory test.
• W5 — EV charger sessions. 2,000 topics, SESSIONWINDOW(ss, 10, 2). Neither Telegraf nor Redpanda Connect has a session window, so they can't express it at all.

Proofs, not vibes. For each engine, workload and rate (5k, 20k, 50k, 100k msg/s), we warm up until the subscription is provably live, send for 30 seconds on a fixed schedule, drain until the sink file stops growing, then verify the output exactly:

• W1: the count of unique message IDs carrying the run tag must equal the closed-form expected filtered count, with no duplicates.
• W2, W4, W5: the sum of per-device counts across all output windows must equal the messages sent, and every device must appear.
• W3: output rows must equal messages sent, and all 10,000 topics must appear.

A step is complete only when its proof holds. "Loss" is the relative shortfall against the proof. This is the part that makes the numbers trustworthy — and, as you'll see, it's the part that caught our own bug.

Results

Nobody else finished the range

Highest tested rate with complete, correct output:

rekuiper completed all 20 steps. Because 100,000 msg/s was the top of the range, we never reached its limit — at 100k on the wide ESPHome workload it used 96.6% of the core, and the windowed workloads used 78–85%, so there's headroom left. eKuiper was solid and complete up to 20,000 msg/s across the board. Telegraf managed 50,000 on two stateless workloads but never produced complete per-device windows at any rate. Redpanda Connect reached 20,000 on stateless workloads and 5,000 on windows.

The memory gap

This is the result we'd frame and put on the wall. Peak engine heap (cgroup anonymous memory) at 20,000 msg/s, the highest rate every engine could still be compared at:

(† marks a step whose correctness proof failed — the memory figure is real, but the engine wasn't producing complete output.)

On the windowed workloads (W2, W4, W5), eKuiper's heap ran to 536–886 MB and Redpanda Connect's system_window pattern — which holds every message of a window before aggregating — hit the 1 GB ceiling. rekuiper stayed at 5–10 MB at every rate on every workload. Two orders of magnitude, on the exact workload edge fleets generate.

CPU: roughly half

At 20,000 msg/s, rekuiper used 44–49% of one core. eKuiper used 86–99%, and the two Go pipeline tools were similar or worse (on the steps where they were even producing correct output). About half the CPU per message, which on a shared single-core box is the difference between comfortable headroom and being one traffic spike away from falling behind.

Where the difference actually comes from

It would be easy, and wrong, to write this up as "Rust beats Go." The language helps, but the CPU difference on the stateless workloads is roughly 2×, not 10×, because MQTT receive, JSON decode and file writing dominate for everyone. The interesting gaps are structural.

Window memory is a design choice, not a language feature. eKuiper's windowed memory grows with message rate — about 125 MB at 5,000 msg/s, 536–886 MB at 20,000 — and hits the 1 GB limit at 50,000, where output loss immediately follows. Redpanda Connect's documented windowing buffers the whole window and reaches the limit from 20,000 msg/s. rekuiper's incremental evaluator keeps one accumulator per device, so heap is a function of fleet size, not traffic. Any of these engines could adopt the same approach; the point is that it's the approach, not the runtime, that matters here.

The loss mechanism is the broker, honestly reported. When an engine falls behind, its MQTT subscription backs up and the broker drops QoS 0 messages for that slow subscriber. This is visible directly in the input counters — for instance, eKuiper received only 1.23 of 3.0 million messages on W1 at 100,000 msg/s. Under QoS 1 the same overload would surface as backpressure on the publishers instead of loss. We report QoS 0 because it's the common, cheap edge default, and because it makes overload measurable rather than hidden.

The bug our own benchmark caught

Here's the part we could have quietly left out, and won't.

An earlier run of this exact harness, on a previous rekuiper build, showed about 15% "loss" on W2 at every rate, and 100% loss with 820 MB of memory at 100,000 msg/s. It looked like overload. It wasn't — it was a correctness defect in our window evaluation.

Our time-window trigger was collapsing the whole window into a single aggregate: it ignored GROUP BY partitioning (emitting one row per window, with group values taken from the first record), ignored WHERE, and buffered and cloned every row on the way. The reason it produced a suspiciously constant shortfall was subtle: the warm-up device's row was absorbing the first window of measured data every time.

Our unit tests didn't catch it, because they aggregated a single group — exactly the case the bug handled correctly. Only the exact per-device proof in the benchmark exposed it. We fixed it (that fix is the incremental evaluator described above) before the final measurements.

We're telling you this because it's the strongest argument in the whole paper for verifying output content: a throughput-only benchmark, or one that counts output rows without checking their identity, would have looked at that defective build and reported it as fast and lean. The bug reduced work by skipping grouping and filtering. Speed without a correctness proof is not a measurement; it's a guess with a stopwatch.

A "neutral" harness detail that reordered the results

One more methodology lesson, because it surprised us.

In an earlier comparison, the output sink file lived on a Windows-drive bind mount, where every write system call is expensive. Telegraf's file output, by default, issues one unbuffered write per metric; Redpanda Connect writes each message individually. Both were bound by the sink, not their own logic — flat CPU around 40–47% while losing up to 89% of messages — purely because of where the file lived. eKuiper was affected too, though less. rekuiper batches its file writes, so it barely noticed.

Moving the sink to the VM's local ext4 filesystem changed the standings substantially. eKuiper W1 at 20,000 msg/s went from 1.7% loss to complete; Telegraf W1 at 20,000 went from 37% loss to complete. Same engines, same rates, same rules — different disk. We kept the superseded runs in the artifact and marked them as such, and the lesson is now a rule we'd apply to any stream-processor benchmark: state where your sink writes and how often it issues system calls, because that detail can quietly decide your rankings.

(There's an honest loose end here too: Telegraf lost a near-constant 9.4% on the windowed workloads at low rates even with a grace period, yet was complete at 50,000 on W4. We didn't find the cause. The configuration is published so someone else can.)

What this doesn't prove

We build rekuiper, so treat the framing with the skepticism it deserves — and here's what to hold against it:

• One host, one repetition. The whole run was a single laptop under Windows 11 and WSL2, one repetition per cell. Host-health snapshots flag several runs as noisy. The gaps are large relative to that noise, and an earlier run showed the same qualitative pattern, but repeated runs on a dedicated Linux host would be stronger.
• A coarse rate ladder. "20,000" means an engine passed 20,000 and failed at 50,000; the true limit is somewhere in between. rekuiper's ceiling wasn't measured at all.
• Defaults, mostly. eKuiper ran with default rule options; a partial run with larger buffers showed a similar pattern but wasn't repeated. Redpanda Connect used its documented windowing pattern — other designs might do better.
• QoS 0 only. No QoS 1/2, TLS, broker reconnect storms, event-time or out-of-order windows, joins, or non-MQTT connectors. rekuiper's MQTT path is its stable, benchmarked path; its other connectors are not yet considered stable.

None of that changes the central, two-order-of-magnitude memory result, but you should know exactly where the edges of the claim are.

Try it, and break it

Everything here is reproducible. The engine, the orchestrator, the load generator, every configuration, and the raw per-step evidence (per-second CPU and memory, generator reports, input counters, host health, image IDs) are published at tag v0.425-beta:

git clone https://github.com/ankur-paan/rekuiper.git
cd rekuiper
# Method, configs and commands:
#   test/benchmark/iiot-mqtt/README.md

A full run — four engines, five workloads, four rates — takes about 1 hour 45 minutes and needs Linux or WSL2 with Docker (cgroup v2), a Rust toolchain, and at least 12 logical CPUs.

If you run IIoT gateways, ESPHome fleets, vehicle telemetry, or EV chargers, the most useful thing you can do with this is try to break it on your own hardware and your own rules — especially ARM, and especially with buffer settings tuned for your traffic. We'd genuinely rather hear where it falls over than where it wins.

Because at the edge, the engine that survives a Monday-morning reconnect storm isn't the one with the biggest throughput number. It's the one whose memory you can still predict when ten thousand devices all start talking at once.

👉 GitHub: https://github.com/ankur-paan/rekuiper

rekuiper v0.425-beta is dual-licensed MIT / Apache-2.0.