Why we stopped using one Elasticsearch index per tenant and built our own

TL;DR: We open-sourced SeaSearch , an Elasticsearch-compatible search engine that keeps index data in S3 and routes ownership instead of replicating data, so tenant count isn't capped by cluster state.

The tax on one index per tenant

If you've built multi-tenant search on Elasticsearch, you've had this argument with yourself.
The clean approach to multi-tenant search is one Elasticsearch index per tenant. Data stays physically separate, queries only hit the relevant tenant's data, and a traffic spike from one customer does not slow down anyone else's.
But every index costs at least one shard, and each shard is basically its own mini Lucene database, with its own files, its own background maintenance work, its own chunk of memory.
Separately, every index's mappings and routing information live in cluster state, which the master maintains and replicates to every node. A few thousand indexes and mapping updates get sluggish. Tens of thousands and the master node becomes the thing you page about at 3am.

We didn't want to write a search engine

Two years ago we started looking for a way out. The first decision was to not build a search engine from scratch. Analyzers, scoring, segment merging, query planning. That's decades of accumulated work and we had no interest in relearning it badly.
ZincSearch got us most of the way. Written in Go instead of the JVM, where Bluge is doing the indexing underneath, and an Elasticsearch-compatible API surface. That handled the runtime footprint and the migration story in one move.
What it didn't solve: running an unbounded number of indexes across a cluster without paying for that isolation in replicated storage and operational complexity. That's the part we built, and it's most of what SeaSearch is.

Put the indexes in S3 and route ownership instead of data

Compute nodes hold no authoritative data, so failover is a map update rather than a data migration.

Four moving pieces:

Compute nodes serve reads and writes. They all talk to the same S3-compatible bucket, where index data is stored.
etcd holds cluster metadata: index metadata and the partition ownership map.
The cluster manager watches node health and decides who owns what.
The proxy (gateway) takes a client request, looks up which node owns that index, and forwards it.

Indexes are hashed into a fixed number of partitions. The cluster manager keeps a partition-to-node map in etcd. The proxy reads that map to route. When a node dies or a new one joins, the manager recomputes the map and hands ownership over.
Nothing moves. The new owner pulls what it needs from S3 on demand.
Compare that to a replica-based cluster, where high availability means a second physical copy of your data on another node, and rebalancing means shipping gigabytes across the network while your cluster is already unhappy. With shared storage, adding read capacity is: start a node, update the map. Losing a node is: update the map.
This is why the index count stops being scary. Creating an index adds an entry to etcd, not a shard to a cluster state document that every node has to agree on.

The obvious objection: S3 is slow

It is. A round trip to object storage is roughly an order of magnitude worse than local NVMe, and search is a latency-sensitive workload. Caching here is mandatory.
The thing that makes the cache easy is that index data is organized into immutable segments. Once a segment is written it can be read or deleted, never modified.
Cache invalidation is the hard part of caching. Immutability deletes the question entirely. If you have the segment locally, your copy is correct.
Compute nodes keep a rotating local disk cache. When it fills, old segments get evicted to make room. That's what lets a node serve an index larger than its own disk: it just churns through segments.

Two things make the cold path tolerable:

Parallel warm-up. Modern datacenter bandwidth is enormous and a single-threaded fetch wastes it. During warm-up we pull many segments from S3 concurrently, which turns a serial latency problem into a bandwidth problem.
Distributed query execution. For very large indexes, SeaSearch can split a single query across multiple compute nodes. Each loads and searches a slice in parallel, and results get aggregated. This does two useful things at once: queries go faster, and cache pressure gets spread out instead of crushing one node.

Latency in practice: the first query after a node starts, or after a segment gets evicted, is slow. Every query after that runs at roughly local-disk speed. For file metadata search, where the active working set is a small fraction of total data, that's been a good trade for two years. If your workload touches all of your data uniformly and constantly, it would be a worse one.

The ranking problem with multiple indexes

Searching across many independent indexes introduces another problem: the relevance scores from those indexes aren't directly comparable.
BM25 scores are computed from index-local statistics: term frequency, document count, average field length. A match for invoice in a twelve-document library gets a score computed against completely different statistics than the same match in a 400,000-document library. Both _search across multiple indexes and _msearch score each index independently.
Merge those results, sort by score, and your top hit mostly reflects which library the file happens to live in. Users don't file that as "inconsistent scoring across index statistics." They file it as "why is this random file at the top."
So we added a unified search endpoint that computes comparable scores across the whole set of indexes:

POST /api/unified_search
{
  "index_queries": [
    { "index": "library-a1b2", "query": {} },
    { "index": "library-c3d4", "query": {} }
  ]
}

The constraint is that the query has to be the same for each index, aside from filters. That's fine for the "search everything I can see" case, which is the case that needed it.

What it doesn't do

Compatibility claims are worth less than a list of what's missing, so here's ours.

No shards or replicas settings. number_of_shards and number_of_replicas are meaningless under shared storage.
Field types are limited to text, keyword, numeric, bool, date, and vector. No nested , object , or flattened .
Mappings are additive. You can add fields. You can't change existing ones.
Unsupported search parameters: indices_boost , knn , min_score , retriever , pit , runtime_mappings , seq_no_primary_term , stats , terminate_after , version .
Your durability is your object store's durability. That's a feature if you're on S3 or a well-run MinIO cluster. It's a problem if you were planning to point it at a single disk and call it a day.

If you're running Elasticsearch for observability with deep aggregation pipelines and ILM policies, SeaSearch is not your replacement and we won't pretend otherwise. If you're running app search for a multi-tenant product, it probably is.

Why we're open-sourcing this

We built SeaSearch because Seafile needed it and we think the architecture solves a problem that other multi-tenant applications may also run into. There's no paid tier and no feature set held behind a commercial license.
We'd rather have people use it, test it, and tell us where it breaks than keep it sitting inside one company's infrastructure.