Behind my own VPS monitoring application there is a small two-node NATS cluster; 2.11.17, in containers, JetStream off. I looked at /varz: in 4 days 13 hours, 3.5 million messages in (8 GB), 5.4 million out (12.9 GB), slow_consumers: 0, 134 subscriptions, max_pending 64 MiB. Because the counter is zero, here is what I had never thought about until today: every one of those 3.5 million messages was carried "at most once". The server does not put the message on disk; if the buffer towards a subscriber fills up it cuts the connection, and what was in the buffer goes with it. Zero slow consumers does not mean zero loss; it means no slow consumer that the server measured.
This behaviour of NATS is documented and deliberate, but "slow consumer" is detected in two separate places, with separate limits and separate outcomes; JetStream is the third answer to the same problem. I measured all three on the same server, with the same 1 KiB messages.
Three limits
In core NATS a slow consumer is caught in two places. In the client library: if the per-subscription pending message/byte limit is exceeded, the library drops the message, hands the application an error event, and the connection stays open. In nats-py this limit defaults to 524,288 messages and 128 MiB. On the server: if the bytes waiting to be written to a connection exceed max_pending (64 MiB by default, MAX_PENDING_SIZE) or the write_deadline (10 s) expires, the server cuts the connection; the documentation explains this as protecting "itself and the integrity of the messaging system". The client reconnects, the messages in between are gone. In JetStream the message is written to disk first, the consumer pulls as much as it can, and what it does not acknowledge the server redelivers after AckWait; no loss, but delay.
The lab
On the server, in a separate Docker network, two nats:2.11-alpine: one plain, one with -js. The publisher is nats-py sending 200,000 messages, each a 12-byte sequence number plus 1 KiB of padding; the subscriber uses the same library; the only things that change from run to run are the processing time and the pending limits. Publisher and subscriber are separate containers, separate processes. The loss count comes from the set of sequence numbers received.
First the ceiling. With no subscriber, nats bench pub "sent" 200,000 messages at 388,762 messages per second, 380 MiB/s: since nobody was listening they all fell on the floor at the server and no counter called it an error; in core NATS a subject without subscribers is /dev/null. Then a fast subscriber: 200,000 published, 200,000 received, in 1.4 seconds.
What the client drops
I put await asyncio.sleep(0.001) into the subscriber's callback: one millisecond per message, without blocking the event loop. I deliberately kept the pending limits small, 65,536 messages and 64 MiB:
published 200000 in 2.40s
received=9502 unique=9502 expected=200000 lost=190498
client_errors=134862 sample=['nats: slow consumer, messages dropped subject: test, sid: 1 ...']
in_msgs=200000
The numbers add up: 134,862 dropped + 65,138 accepted = 200,000; the client read all of them off the socket. The binding limit was not the message count but the byte limit: about 64,800 messages of 1,036 bytes fit into 64 MiB, and from that point the library dropped every incoming message and raised an error event. The application managed to process 9,502 messages in 25 seconds; when time ran out, 55,636 messages were still in the library's queue, not dropped. On the server side slow_consumers is still zero, because from the server's point of view everything went fine: the socket was read, the buffer drained. Only code that listens to the client's error event sees the loss.
I repeated the experiment with very large limits (10 million messages, 8 GB): again ~10,000 messages processed in 25 seconds, but in_msgs=200000 and zero errors; all 200,000 messages were waiting in the client's memory, at least 207 MB of data, more with Python object overhead (I did not measure the RSS). The limit is the trade-off between loss and memory; remove it and instead of loss the RSS grows, and nobody says anything until the OOM.
What the server cuts
To reach the server's limit the client has to stop reading the socket. I put time.sleep(0.002) into the callback, a blocking sleep: when the event loop stops, the library's reader task stops too, the socket buffer fills, TCP slows the other side down, and the output queue the server keeps for that connection grows; the counterpart, one floor up, of the kernel queue in yesterday's post, this time with the application's own max_pending counter. 300,000 messages:
published 300000 in 7.37s
received=5964 unique=5964 expected=300000 lost=294036
client_errors=1 sample=['nats: unexpected EOF']
[INF] 172.18.0.4:53914 - cid:19 - Slow Consumer Detected: MaxPending of 67108864 Exceeded
"slow_consumers": 1
When its 64 MiB output queue filled, the server closed the connection; on the client this appeared as unexpected EOF, the library reconnected, but the publisher had long finished. 5,964 messages processed, 294,036 gone. The publisher's 7.37 seconds is striking too: with 1.5 times the messages I would have expected 3.6 from the first experiment's 2.4. The server has a mechanism for this, a "stall" gate that pauses a publisher for at most 10 ms per read loop once a consumer's buffer passes 75 percent of max_pending (a "Producer was stalled" warning above 5 ms; the main branch adds a stalled_clients counter to /varz, 2.11.17 does not have that field). I did not measure it, so I do not attribute the cause to it; I only note it. slow_consumers, however, became 1 for the first time. This is the meaning of the zero in my cluster: my subscribers always read the socket in time; how many messages were dropped inside the library the server does not know, and there is no such counter.
What JetStream holds
I moved the same blocking consumer to JetStream: a file-based stream called LAB, 50,000 messages via js.publish (each waiting for the server's ack), then pull_subscribe in batches of 100, with time.sleep(0.002) and an ack per message:
js.publish 50000 msgs in 92.50s = 541 msg/s (each waits for ack); stream messages=50000
received=38300 unique=38300 lost=0 redelivered=0 in 120.1s
consumer pending=11700 ack_pending=0
At my 120-second time limit 38,300 messages had been processed, zero lost; the remaining 11,700 are in the stream, in the consumer's pending, waiting for the next fetch. The consumer's slowness does not concern the server, because in the pull model the consumer asks for the message; the server writes at most the requested batch at a time (100 messages here), the 64 MiB output queue never fills, no connection ever qualifies for cutting.
I also poked at the acknowledgement mechanism. I opened a consumer with max_ack_pending=10, ack_wait=5, fetched 10 messages and acknowledged none:
first fetch: [1..10] delivered: [1,1,1,1,1,1,1,1,1,1]
second fetch immediately: timeout (max_ack_pending=10 reached, nothing acked)
fetch after ack_wait 5s: [1..10] delivered: [2,2,2,2,2,2,2,2,2,2]
after ack: [11..20]
With 10 unacknowledged messages outstanding the second fetch came back empty; the server suspends delivery at the MaxAckPending limit; the documentation says that for push consumers this is "the only form of flow control" (there is also a separate FlowControl option), while for pull consumers the request itself provides implicit flow control. Five seconds later the same 10 messages came back, num_delivered 2; after acking, 11-20. No loss, but there is a price: a consumer that cannot finish processing receives the same message again and again, and the application has to tell them apart (idempotent processing keyed on num_delivered and the stream sequence; the publisher-side Nats-Msg-Id deduplication is a separate mechanism). This is the "least" in at-least-once.
The price
With nats bench on the same machine:
Core pub (no subscriber) 388,762 msg/s
JetStream pub sync 826 msg/s (each message waits for an ack; 541 with nats-py)
JetStream pub async, batch 500 22,470 msg/s
JetStream fetch (explicit ack) 47,683 msg/s
Between synchronous publish and core there is a factor of 470; the difference is every message waiting for the server's ack in turn: one message in flight at a time. Even though the disk path is the same, asynchronous batched publish brings the factor down to 17, and that is the path used in production. "No loss" has a footnote too: on a single-replica (R1) file store the default sync_interval is 2 minutes; if the operating system crashes, acknowledged messages from the last two minutes can be lost, the remedy being sync_always (which drops throughput to a few hundred msg/s) or R3. The consumption side is 47 thousand/s. These numbers are for one machine, one node, file storage and 1 KiB messages; on a clustered (R3) stream every write also waits for a quorum ack and the figure drops. The gist of the comparison does not change: core NATS is fast because it remembers nothing.
For my own cluster
After this measurement I read the zero in /varz differently. For monitoring data, at-most-once is usually the right choice: if a metric sample is lost the next one comes, better five seconds stale than a bloated memory. But if you carry commands or events over the same cluster, which of "the server cuts" and "the client drops" hits you is decided by your client library's limits, not by the server's counter. For me the order is this. Always log the client's slow consumer error event, because the server does not know. Scale a slow subscriber out with a queue group, as the documentation suggests; raising max_pending delays the loss and pays with memory. If loss is unacceptable, move the subject into a stream, build the consumer in the pull model, choose AckWait and MaxAckPending from the processing time, recognise a redelivered message, and do not forget the stream's own limits (MaxMsgs, MaxBytes, DiscardPolicy): a full stream either drops the old or rejects the new. Watch slow_consumers in /varz and the "Producer was stalled" lines in the log, but do not conclude "no message was dropped" from zero; that counter only counts the connections the server cut.
Measurements in Docker on Ubuntu 24.04, nats:2.11-alpine (nats-server 2.11.17), natsio/nats-box (nats CLI 0.4.0), nats-py 2.10.0, Python 3.12; one machine, one node, file storage. Server defaults from server/const.go (MAX_PENDING_SIZE, DEFAULT_FLUSH_DEADLINE, DEFAULT_PING_INTERVAL), library defaults from the installed package (defined in nats/aio/subscription.py) (DEFAULT_SUB_PENDING_MSGS_LIMIT=524288, DEFAULT_SUB_PENDING_BYTES_LIMIT=134217728). My monitoring cluster's /varz values from the evening of 16 September 2026.
Official Sources
- NATS docs — Slow Consumers: client drops, server cuts, write_deadline
- NATS docs — Core NATS: best-effort, at-most-once
- NATS docs — JetStream: at-least-once, exactly-once, double ack
- NATS docs — Consumers: AckWait, MaxAckPending, BackOff
- nats-server server/const.go — MAX_PENDING_SIZE 64 MiB, ping interval
- nats-server server/client.go — "Slow Consumer Detected: MaxPending of %d Exceeded"
- nats.py — nats/src/nats/aio/subscription.py: subscription pending limits
- NATS docs — Subjects: messages without subscribers are discarded
- natscli — nats bench
- Docker Hub — official nats image (2.11-alpine used)
