Cost Optimization for Log Storage with ILM and Tiering

Operational symptoms are clear: bills spike after bursts, queries over older windows time out, shard counts grow, operator toil increases, and auditors ask for older evidence that you can’t quickly find. Those are not abstract problems — they’re the cost-performance, compliance, and availability trade-offs you accept when every log is treated the same.

Contents

• How hot/warm/cold tiers cut costs — and what you trade for speed
• Modeling retention by use case: SRE, security, compliance, and analytics
• Exact ILM policy patterns that save money (with cURL and JSON examples)
• Sizing shards, compression and storage knobs that reduce GBs and bills
• Cold archiving, searchable snapshots, and compliance-safe retention
• Actionable Runbook: ILM, tiering and retention checklist you can run tonight

How hot/warm/cold tiers cut costs — and what you trade for speed

The simplest cost lever is storage class: place the small fraction of data you query frequently on fast, expensive media and push everything else down the stack. In Elasticsearch terms that becomes the hot, warm, cold, and (optionally) frozen tiers, and you orchestrate movement with index lifecycle management (ILM). ILM automates rollover, phase transitions, and deletion so policy — not manual ops — controls cost and risk.

Quick definitions and trade-offs:

• Hot — small-write, low-latency tier (NVMe/SSD), the write path and recent-search tail. Keep indices that are actively written or queried here. Higher $/GB, fastest queries.
• Warm — denser nodes or cheaper SSD/HDD, where you do read-heavy retrospectives and retention optimizations (shrink, forcemerge). Moderate $/GB, moderate query latency.
• Cold — backed by object storage via searchable snapshots or cold node roles; indices are rarely queried but remain searchable. Lowest ongoing cost for indexed searchability, but query latency and mount costs can increase.
• Frozen — partially-mounted searchable snapshots for very deep lookbacks with minimal cluster footprint (higher per-query latency).

Tier actions you’ll use in ILM: rollover, forcemerge, shrink, allocate/migrate, searchable_snapshot, freeze/unfreeze (depending on ES version), and delete. Use rollover to control shard sizes and searchable_snapshot on the cold tier to offload storage to object repositories.

Important: searchable snapshots usually reduce cluster storage and remove the need for replicas, but they can be more expensive in environments where snapshot repository reads or cross-region transfer costs are high. Validate repository read/egress costs before wholesale adoption.

Modeling retention by use case: SRE, security, compliance, and analytics

You must design retention against use cases. Treat retention as a product decision: every day you keep logs costs money; every day you delete them risks missing investigations. Classify your streams and assign policies.

Common log classes and sample retention patterns (start conservative — measure — tighten):

• Operational troubleshooting / SRE: short, high-fidelity, high-query-frequency. Keep 7–30 days in hot/warm (fast search), then move to cold if needed.
• Security/forensics: moderate-term quick-search (90 days hot/warm) and long-term archive (1–7 years) for deep investigation and regulatory holds.
• Compliance / audit trail: governed by policy — often multi-year — kept in immutable archives or object-store snapshots with legal holds.
• Business analytics or metrics-derived logs: downsample or transform to metrics after a short high-fidelity window, then archive raw events to cold/object store or delete.

A compact cost model (steady-state view):

• Variables:
  • I = ingest rate (GB/day)
  • R = retention days for the stream
  • C = post-ingest compression factor (fraction of raw size; e.g., 0.5)
• Steady-state storage for the stream (GB) = I * R * C
• Monthly cost for the stream = sum_t (storage_in_tier_t_GB * price_per_GB_month_t)

Example (illustrative numbers only — replace with your invoices):

• Ingest I = 100 GB/day, C = 0.5 → effective 50 GB/day stored
• Retention: 7d hot, 23d warm, 335d cold → total 365 days
• Steady-state storage = 50 GB/day * 365 = 18,250 GB (~17.8 TB)
• If cold object-store price ≈ $0.00099/GB-month (S3 Glacier Deep Archive example), warm ≈ $0.04/GB-month (hypothetical), hot ≈ $0.12/GB-month (hypothetical) you can compute per-tier spend. Use your actual node costs or cloud disk invoices for accurate warm/hot prices.

Why a steady-state model? Because once you reach a stable ingest rate and retention policy, your total stored GB is constant and monthly storage costs are predictable. Measure ingestion and compression carefully using the API and Metricbeat to get I and C.

Exact ILM policy patterns that save money (with cURL and JSON examples)

Here are pragmatic ILM patterns proven in production. Use a canary dataset before rolling cluster-wide.

1) Register a snapshot repository (S3 example)

# assumes repositories-s3 plugin or cloud provider support; prefer IAM role for production
curl -X PUT "https://es.example:9200/_snapshot/my_s3_repo" -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "my-company-es-snaps",
    "region": "us-east-1"
  }
}
'

Registering a repository lets searchable_snapshot mount snapshots from that repo. Use IAM roles or the keystore for credentials.

2) Create a conservative ILM policy that rolls, compacts, moves, and snapshots

curl -X PUT "https://es.example:9200/_ilm/policy/logs-ilm-policy" -H 'Content-Type: application/json' -d'
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb",
            "max_age": "7d"
          },
          "set_priority": {"priority": 100}
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1,
            "index_codec": "best_compression"
          },
          "shrink": {
            "number_of_shards": 1
          },
          "allocate": {
            "require": {"data": "warm"}
          },
          "set_priority": {"priority": 50}
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "my_s3_repo"
          },
          "allocate": {
            "require": {"data": "cold"}
          },
          "set_priority": {"priority": 0}
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": {
          "wait_for_snapshot": {"policy": "daily-snapshots"},
          "delete": {}
        }
      }
    }
  }
}
'

Notes on the policy:

• rollover keeps shard size in the target range (shard-sizing guidance below).
• forcemerge with index_codec: best_compression can reduce storage; this happens in warm where write pressure is low.
• searchable_snapshot in the cold phase mounts the snapshot and allows you to remove replicas and reduce node count. Test repository-read costs first.

3) Index template and write alias

curl -X PUT "https://es.example:9200/_index_template/logs-template" -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "index.lifecycle.name": "logs-ilm-policy",
      "index.lifecycle.rollover_alias": "logs-write",
      "index.number_of_shards": 1,
      "index.codec": "best_compression"
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "host":       { "type": "keyword" },
        "message":    { "type": "text", "index": false } 
      }
    }
  },
  "priority": 200
}
'

Create the initial write index:

curl -X PUT "https://es.example:9200/logs-000001" -H 'Content-Type: application/json' -d'
{
  "aliases": {
    "logs-write": { "is_write_index": true }
  }
}
'

Make sure the rollover_alias and templates are in place before you start production ingestion so ILM applies automatically.

4) Create SLM (snapshot lifecycle management) to keep retention-controlled snapshots

curl -X PUT "https://es.example:9200/_slm/policy/daily-snapshots" -H 'Content-Type: application/json' -d'
{
  "schedule": "0 30 1 * * ?", 
  "name": "<daily-snap-{now/d}>",
  "repository": "my_s3_repo",
  "config": { "indices": ["logs-*"], "include_global_state": false },
  "retention": { "expire_after": "90d", "min_count": 5, "max_count": 180 }
}
'

Use SLM for backup retention and coordinate ILM wait_for_snapshot if you require on-disk snapshots before deletion.

Sizing shards, compression and storage knobs that reduce GBs and bills

Storage reduction is a combination of fewer shards, better compression, and reducing redundant copies where appropriate.

Shard sizing and management

• Target an average shard size in the range of tens of GBs — commonly 20–40 GB per shard for time-series indices is a practical target. Too many small shards costs CPU/heap; too-large shards increase recovery time. Always benchmark your own queries.
• Use rollover to control shard growth; use shrink in warm to reduce primary shard count for old, read-only indices.
• Track shards-per-node ratio — modern ES reduced heap pressure per shard, but keep the total shards per node well below limits recommended for your Elasticsearch version and heap size.

Compression and mapping

• Set index.codec: best_compression (ZSTD/DEFLATE or best_compression) on read-only indices to reduce stored bytes at the cost of CPU when reading; apply it at forcemerge time in warm phase. Experiments show meaningful storage savings for logs with repeated metadata fields.
• Remove unnecessary _source fields or use index.mapping.source.mode: synthetic where appropriate to reconstruct source from doc_values (careful: this affects retrieval patterns). Use doc_values and disable indexing for fields you never search on to reduce inverted index overhead.
• When you must keep raw events but do not need per-document retrieval, consider downsampling (rollups) or storing aggregates and archiving raw events to searchable snapshots.

Forcemerge strategy

• forcemerge to 1 segment for indices that are no longer written can reduce footprint and speed certain searches — but it’s resource-intensive. Run merges in warm hardware during off-peak windows and throttle/monitor the force-merge queue.

Practical knobs list (short):

• index.lifecycle.rollover_alias + max_primary_shard_size (rollover by size)
• forcemerge with index_codec: best_compression in warm
• shrink to reduce primaries after write window
• searchable_snapshot in cold to move to object store and remove replicas

Cold archiving, searchable snapshots, and compliance-safe retention

Searchable snapshots let you keep data in cheap object stores while remaining able to search it — a potent cost control. They mount snapshots from your snapshot repository and typically eliminate the need for replica shards for those indices, lowering cluster disk requirements.

How searchable snapshots fit into ILM:

• Use searchable_snapshot in the cold or frozen phase of ILM and specify the snapshot_repository. ILM will mount the snapshot and replace the managed index with a searchable snapshot index.
• If you need guaranteed immutable evidence for audits, combine snapshots with object-store-native retention/WORM features (e.g., S3 Object Lock for AWS) and use SLM to manage snapshot lifetimes.

ILM + SLM interplay:

• ILM wait_for_snapshot lets you ensure an SLM policy ran a snapshot before ILM deletes an index. This is a common compliance pattern: snapshot → searchable snapshot mount → ILM delete after snapshot retention ensured.

Compliance considerations

• Regulatory retention durations and immutability requirements differ across jurisdictions and standards. Use snapshots + object-store locking (S3 Object Lock or equivalent) where a compliance-grade WORM is required. Plan your snapshot retention rules and S3 bucket/object lifetime accordingly; test restore and legal-hold workflows.
• Keep an auditable trail of snapshot creation/deletion and secure the SLM and repository credentials.

Actionable Runbook: ILM, tiering and retention checklist you can run tonight

This is a runbook you can execute in stages. Each step is concrete and minimal-risk.

1. Inventory and measure (day 0)

   • Identify top-5 heavy producers (GB/day) and top-10 heaviest indices using:

    # quick health and store sizes
    curl -s "https://es.example:9200/_cat/indices?v&h=index,docs.count,store.size,ilm.policy,ilm.phase"
   

• Collect ingestion rate and compression factor: run Metricbeat or use GET _nodes/stats/indices and average indexing.index_total over 24–72 hours.

1. Classify (day 0–1)

   • Tag each stream: hot-only (debug), hot+warm (ops), security, compliance, analytics. Decide initial retention buckets (e.g., 7/30/365 or 90/365/1825).

2. Build SLM & snapshot repo (day 1)

   • Create an S3 (or provider) snapshot repository and an SLM policy for daily snapshots; validate successful snapshots and retention with GET _slm/stats and GET _snapshot/my_s3_repo/_all.

3. Pilot ILM on one low-risk stream (day 2–7)

   • Create a logs-ilm-policy (similar to the example earlier), apply it via a template.
   • Create a canary index (logs-canary-000001) with alias, ingest a small sample, and observe lifecycle transitions:

    curl -s "https://es.example:9200/_ilm/explain?index=logs-canary-000001"
   

• Validate forcemerge, shrink, and searchable_snapshot steps and measure query latencies for cold mounts.

1. Observe metrics and tune (week 1–2)

   • Key metrics to watch (API / Metricbeat): | Metric | API / Where | Why watch | Example alert | |---|---:|---|---| | Indexing rate (docs/s, GB/s) | Metricbeat index / _nodes/stats/indices | Ingest spikes that break rollovers | > baseline * 2 for 1h | | Store size per index | _cat/indices h=store.size | Tracks tiering and shrink effectiveness | sudden daily growth >10% | | Shard count per node | _cat/shards / Metricbeat | Oversharding => heap pressure | > configured shards/node limit | | ILM errors | _ilm/explain | Policy application and failures | any failed_step | | SLM failures | _slm/stats | Snapshot success and retention | failed snapshot count > 0 |
   • Tune min_age and max_primary_shard_size to match your ingestion and query patterns. Use alerts to capture failed ILM/SLM actions.

2. Validate restore and query paths (week 2)

   • Perform a restore from searchable snapshot and measure end-to-end time. Confirm your analysts can run the queries they need within required SLAs.

3. Rollout and incremental tightening (week 3+)

   • Expand to another 10 datasets. Recalculate cost delta between baseline and optimized policy.
   • Reassess high-query older streams; some must remain hot/warm even if costly.

Troubleshooting commands

• Check ILM progress and failures:

  curl -s "https://es.example:9200/_ilm/explain?pretty"

• Check SLM status:

  curl -s "https://es.example:9200/_slm/stats?pretty"

• See snapshot repository content:

  curl -s "https://es.example:9200/_snapshot/my_s3_repo/_all?pretty"

Operational guardrails

• Start with low-risk datasets and limit how many indices can transition in parallel to avoid force-merge queues.
• Use replicate_for option with searchable snapshots to temporarily add a replica for a short window if query volume demands, then let ILM remove it.
• Always test the cost profile in your environment — object-store egress/GET costs and region egress can flip economics quickly.

Sources:
Index lifecycle management (ILM) in Elasticsearch - Official ILM overview and API; details on phases, rollover, and when to use ILM.

Searchable snapshots - How searchable snapshots work, their cost/replica trade-offs, and ILM integration.

How many shards should I have in my Elasticsearch cluster? - Practical shard-size guidance (commonly ~20–40 GB shard target for time-series).

Save space and money with improved storage efficiency in Elasticsearch 7.10 - Details on compression choices and storage efficiency improvements (e.g., best_compression).

Amazon S3 Pricing - Official S3 storage-class pricing and retrieval/transition notes (useful for modeling searchable-snapshot repository costs).

Index lifecycle actions - Reference of available ILM actions like forcemerge, shrink, allocate, and searchable_snapshot.

Create, monitor and delete snapshots (Snapshot lifecycle management SLM) - How to automate snapshot creation and retention with SLM and integrate with ILM.

Collecting monitoring data with Metricbeat - Which metrics to collect and how to use Metricbeat for Elasticsearch monitoring.

S3 repository (snapshot/restore) - How to register an S3 snapshot repository and recommended settings (IAM, keystore usage).

doc_values - Explanation of doc_values, when to disable them, and mapping strategies to reduce disk usage.

S3 Object Lock – Amazon S3 - S3 Object Lock (WORM) and retention modes for compliance-oriented archival.

Execute the runbook, measure ingestion and storage before and after each change, and rely on ILM as the control plane that turns retention policy into predictable cost.