In Part 1: Economics & System Design, we established our cloud economics and architecture blueprint. In Part 2: Ingesting the Solana Firehose with Rust, we built a resilient, zero-copy Rust ingestion service that decodes raw transactions and pumps structured swap events into NATS.
Now, we have to solve the final piece of the puzzle: Storage and Real-Time Delivery.
Piping millions of events into a traditional relational database like PostgreSQL will instantly lock tables, bloat disk space, and crash under heavy read/write concurrency. To analyze billions of historical swaps while simultaneously pushing live market feeds to end users with sub-second latency, we need a two-tier consumption layer: ClickHouse for analytical storage, and Axum WebSockets for live event fanout.
01. Consuming NATS JetStream & ClickHouse Schema Design
When consuming high-velocity data streams, database maintenance, schema migrations, or unexpected container restarts are inevitable. If your consumer reads directly from an in-memory queue, restarting your database means dropping millions of incoming swap events.
NATS JetStream: Persistence as a Safety Net
To guarantee zero data loss, we configure NATS with JetStream persistence. JetStream writes incoming messages to an encrypted, distributed disk stream. If our database goes offline for 15 minutes, NATS retains the stream and seamlessly replays every missing message the moment the database re-establishes its connection.
💾 JetStream vs. core NATS: Core NATS is fire-and-forget. JetStream adds durable, persistent, replayable streams with consumer acknowledgements. For a financial data pipeline, JetStream is non-negotiable. A consumer crash with core NATS means a data gap; with JetStream it means a brief lag before catch-up.
The ClickHouse Schema
For storage, ClickHouse is the clear winner. As a column-oriented DBMS, ClickHouse compresses data by up to 80% and can execute aggregate queries across billions of rows in milliseconds — the exact performance profile a swap analytics dashboard demands.
-- Production ClickHouse Schema for Solana Swaps
CREATE TABLE IF NOT EXISTS solana_swaps
(
signature String,
slot UInt64,
block_time DateTime CODEC(DoubleDelta, ZSTD),
program_id LowCardinality(String),
signer String,
token_in String,
token_out String,
amount_in UInt64,
amount_out UInt64,
created_at DateTime DEFAULT now()
)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(block_time)
ORDER BY (program_id, token_in, token_out, block_time)
SETTINGS index_granularity = 8192;
By using the MergeTree engine, partitioning by month, and selecting an ordering key optimised for token pair filtering — (program_id, token_in, token_out, block_time) — ClickHouse skips 99% of the disk data during typical analytical queries.
⚡ Key insight: Using specialised codecs like DoubleDelta and ZSTD on timestamp columns dramatically reduces disk footprint, allowing billions of swap records to run on low-cost NVMe storage. LowCardinality(String) on program_id further reduces the column's on-disk size by up to 90%.
- High-Throughput Batch Ingestion The single most common mistake engineers make with ClickHouse is executing single-row INSERT statements for every incoming event. ClickHouse is designed for bulk writes; single-row inserts create millions of tiny data parts on disk, triggering severe write amplification and bringing the cluster to its knees.
Dual-Threshold Flushing with tokio::select!
Our Rust consumer service pools incoming NATS messages into memory buffers and executes bulk inserts using two thresholds:
Threshold 1 (Batch size): ≥ 10,000 rows → flush immediately
Threshold 2 (Time interval): every 1.0 s → flush if buffer non-empty
tokio::select! races both futures concurrently. Whichever fires first triggers the flush. This means during peak volume the pipeline flushes at 10k-row boundaries (maximising throughput), and during quiet periods it flushes every second (minimising latency for dashboards).
use clickhouse::Client;
async fn run_clickhouse_consumer(
mut nats_sub: JetStreamSubscription,
client: Client,
) {
let mut buffer = Vec::with_capacity(10_000);
let mut interval = tokio::time::interval(Duration::from_secs(1));
loop {
tokio::select! {
// Priority 1: Fill buffer from the NATS stream
Some(msg) = nats_sub.next() => {
if let Ok(swap) = bincode::deserialize::<DecodedSwap>(&msg.payload) {
buffer.push(swap);
msg.ack().await.ok();
}
if buffer.len() >= 10_000 {
flush_to_clickhouse(&client, mut buffer).await;
}
}
// Priority 2: Flush periodically during low volume
_ = interval.tick() => {
if !buffer.is_empty() {
flush_to_clickhouse(&client, mut buffer).await;
}
}
}
}
}
By decoupling row-by-row arrivals into structured bulk inserts, we maintain near-zero CPU usage on the database cluster while keeping end-to-end swap visibility latency under one second at all volume levels.
03. The Real-Time Interface: Decoupled WebSocket Fanout
Frontend applications — live trading dashboards, arbitrage bots, portfolio trackers — need to see swaps the exact millisecond they happen. Polling ClickHouse repeatedly to check for new rows creates unnecessary read pressure on your analytical storage and introduces artificial latency.
Instead, we build a dedicated, lightweight WebSocket service using Axum that listens directly to the NATS subject — completely independent of the ClickHouse consumer.
Helius RPC ──> Rust Ingester ──> NATS JetStream ──┬──> ClickHouse (Batch / Analytical)
└──> Axum WS (Live Fanout / Clients)
Because NATS supports pub/sub topics out of the box, our Axum WebSocket service acts as a pure fanout router:
- When a client connects to
wss://api.xroot.dev/ws/swaps?pair=SOL-USDC, Axum subscribes them to the corresponding NATS subject (e.g.solana.swaps.raydium). - When the Rust Ingester publishes a decoded swap to NATS, NATS broadcasts it simultaneously to all active WebSocket clients and the ClickHouse batch consumer — no polling, no secondary query.
- Thousands of concurrent frontend WebSocket connections consume live data feeds with zero impact on database performance, since ClickHouse is never touched by the fanout path.
🔀 Why not query ClickHouse for live data? ClickHouse excels at analytical aggregation — it is not designed for sub-millisecond single-row lookups. Routing live streaming clients through NATS keeps the database free for the heavy aggregation workloads it was built for: hourly volume, OHLCV candles, top-pair rankings.
Conclusion & System Summary
We have successfully engineered an end-to-end, high-throughput Web3 data pipeline from scratch. Here is the complete picture:
- Filtered RPC Ingestion: Optimised Helius WebSocket subscriptions cap costs at under 8M credits/day on a $999/month budget — no wasted credit burn.
-
Zero-Copy Rust Engine:
carbon+ boundedtokiochannels on a cost-effective 4-CPU pod handle thousands of events per second with no GC pauses and no memory leaks. - Guaranteed Delivery: NATS JetStream persists the stream to disk, ensuring zero data loss during downstream outages and enabling seamless replay on reconnect.
- Analytical Storage & Fanout: A partitioned ClickHouse MergeTree table delivers sub-second analytical queries over billions of rows. A decoupled Axum WebSocket service fans out live swap events to frontend clients with zero database pressure.
