Hyperliquid's public S3 archive (hyperliquid-archive, requester-pays)
ships historical market data as .lz4-compressed CSVs. The obvious path
is pip install lz4 and move on — except on a locked-down box, or inside
a Lambda, or anywhere you don't control the runtime, pulling in a
compiled dependency for one decompression step is often more friction
than it's worth. So I wrote a pure-Python LZ4 Frame decoder: no C
extension, no pip install, just stdlib.
The frame format, in short. An LZ4 frame starts with a 4-byte magic
number, a flags byte, a block-descriptor byte, then a sequence of
blocks — each one either raw (top bit of the 4-byte size header set) or
LZ4-compressed, terminated by a 4-byte zero. Nothing exotic; the
complexity is entirely in decoding the compressed blocks themselves.
The block format is a token-driven copy machine. Each sequence
starts with one token byte: the high nibble is a literal-length count,
the low nibble is a match-length count. If either nibble reads 15, that
signals "read more length bytes, one at a time, while each byte equals
255" — LZ4's variable-length integer encoding. After the literals comes
a 2-byte little-endian offset (how far back to copy from), then the
match itself, copied byte by byte from output[-offset:] — which has
to run byte-by-byte rather than in a single slice, because pathological
inputs can have the source and destination ranges overlap inside the
match (part of what makes LZ4 compress repeating patterns so well).
What actually caught bugs, not the happy path: the last sequence in
a block is literals-only, no match — trying to read a match after the
final literal run reads past the end of the block. And a compressed
block can validly decode to more or less than the uncompressed block
size the frame header implies, which stops being a bug and starts being
"expected" once you read the spec closely enough.
I verified it against a synthetic frame (one raw block + one compressed
block, block-independence flag set) round-tripping to the exact original
bytes — not against a real Hyperliquid file, since correctness at the
byte level doesn't depend on which archive produced the input.
The other half of the tool generates the aws s3 commands for the
three requester-pays datasets that matter here — asset_ctxs,
market_data (filtered by coin and date), and hl-mainnet-node-data's
node_fills_by_block — and optionally pipes decompressed CSV straight
into DuckDB for a one-line Parquet conversion, if you have DuckDB
installed. Neither of those needs the decoder to be exposed to build on
top of; they just call it.
Full CLI (decoder + S3 command generator + Parquet conversion):
https://theglitchlist.com/product/lz4-to-parquet-converter-cli-s3-sync/?utm_source=devto&utm_medium=article&utm_campaign=n09_lz4
