Hello, fellow developers! 🧑💻
When building a modern B2B SaaS ecosystem, bulletproof observability isn't a luxury—it's a strict survival requirement. If you are handling distributed cron jobs, processing thousands of webhook deliveries, or managing isolated services, you need to know exactly what happens under the hood when things go wrong.
While architecting the infrastructure for Verne Software, I decided to take complete control of our telemetry data by self-hosting SigNoz. The official repository makes it look incredibly easy: just run a docker-compose up script and you are supposedly good to go. You think: "Great, a 10-minute setup and I can get back to writing business logic." Well, not quite.
The reality is that official documentation often lags behind major architectural shifts in underlying Docker images. What was supposed to be a quick deployment turned into a deep dive into ClickHouse configurations, OpenTelemetry bugs, and networking quirks.
In this article, I want to share my recent battle scars. We will go through the specific traps I encountered while trying to get the modern SigNoz stack running in an isolated Docker network—and exactly how to fix them.
The Target Architecture (The "Happy Path")
Before we dive into the errors, let's look at the final, working setup. The core principle here is strict network isolation and a highly deterministic boot sequence.
Network Isolation
The entire SigNoz stack lives inside an isolated Docker network (verne_observability). The SigNoz API and UI container is additionally connected to an internal network (verne_internal). This allows our core API to proxy dashboard requests without exposing the database or internal collector ports to the outside world.
The Boot Sequence
In the modern stack, you can no longer start everything at once. Order matters immensely. The services must wait for their neighbors to be completely ready:
signoz_init_clickhouse: A one-shot container that downloads essential UDF binaries (like histogramQuantile).signoz_zookeeper: The coordinator for ClickHouse.signoz_clickhouse: The main datastore.signoz_telemetrystore_migrator: A crucial bootstrap step that handles schemas.signoz_otel_collector&signoz: The final data consumers.
Here is how you enforce this dependency chain in your docker-compose.yml to prevent race conditions:
signoz_telemetrystore_migrator:
# ...
depends_on:
signoz_clickhouse:
condition: service_healthy # Wait until DB can accept migrations
signoz_otel_collector:
# ...
depends_on:
signoz_telemetrystore_migrator:
condition: service_completed_successfully # Start ONLY after schema is created
Trap #1: The ClickHouse v25+ Configuration Shift
The Symptom: You mount your custom clickhouse-config.xml exactly as older tutorials suggest, but ClickHouse refuses to start, throwing errors about missing paths or invalid settings at the top level.
The Reality: With ClickHouse 25.5.6, the configuration paradigm has shifted. Replacing the main config.xml outright is a recipe for disaster. Furthermore, certain settings have been strictly recategorized.
For instance, you might see this exact crash loop: Code: 137. A setting 'log_queries' appeared at top level in config. But it is user-level setting that should be located in users.xml inside <profiles> section.
The Fix: Stop mounting over the default config. Instead, utilize the config.d/ and users.d/ directories for your overrides.
If you want to enable query logging, you must define it specifically inside a user profile. Create a users.xml file and mount it to /etc/clickhouse-server/users.d/users.xml:
<clickhouse>
<profiles>
<default>
<log_queries>1</log_queries>
</default>
</profiles>
</clickhouse>
Keep your cluster topologies and UDF paths in config.d/, and your profiles and passwords in users.d/. Respecting this separation will save you hours of debugging.
Trap #2: The Invisible Migrations & Missing Databases
The Symptom: You fire up your stack, and the OpenTelemetry Collector immediately crash-loops, screaming: Database signoz_traces does not exist.
The Reality: In older iterations, you could often get away with starting the datastore and the collector simultaneously. The schemas were either auto-created on the fly or bundled differently. Today, schema creation is strictly decoupled from the application logic. If you just start ClickHouse and the OTel Collector, the necessary databases (signoz_traces, signoz_metrics, signoz_logs) simply will not exist.
The Fix: You need to embrace the one-shot container pattern. Specifically, you must introduce ephemeral containers that do the dirty work before the main services are allowed to boot.
First, as mentioned in our boot sequence, signoz_init_clickhouse must run to download essential User Defined Functions (like the histogramQuantile binary) directly into the ClickHouse user_scripts directory.
Second, and most importantly, you need the signoz_telemetrystore_migrator. This container handles the complex bootstrap process and synchronizes the schemas. Your collector should only run after this migrator exits with a success code.
# The crucial pre-flight check for the collector
signoz_otel_collector:
image: signoz/signoz-otel-collector:${OTELCOL_TAG:-v0.111.10}
command:
- -c
- |
/signoz-otel-collector migrate sync check &&
/signoz-otel-collector --config=/etc/otel-collector-config.yaml
depends_on:
signoz_telemetrystore_migrator:
condition: service_completed_successfully
By explicitly chaining migrate sync check in the command and waiting for the migrator container, you eliminate the race condition completely.
Trap #3: Taming the OpenTelemetry Collector
The Symptom: The Collector fails to start due to configuration parsing errors, missing image tags, or deprecated flags.
The Reality: The OpenTelemetry ecosystem moves incredibly fast, and SigNoz's custom collector is no exception. Between versions, breaking changes are silently introduced, turning perfectly good configuration files into landmines.
The Fixes: Here are the three specific collector headaches you need to watch out for:
Image Tagging Roulette: If you are pinning versions in your .env file (which you absolutely should be doing in production), pay attention to the exact naming convention. Dropping the v prefix (e.g., trying to pull 0.144.2 instead of v0.144.2) will result in an image pull failure. It sounds trivial, but it's a remarkably common stumbling block when updating the stack.
The Renamed Exporter: The telemetry pipeline configuration has evolved. If you are migrating an older otel-collector-config.yaml, you must update the metrics exporter name. The legacy clickhousemetricswrite has been deprecated and replaced. You now need to use signozclickhousemetrics to route your data correctly.
Feature Gates: Clean up your CLI commands. Deprecated flags (like -pkg.translator.prometheus.NormalizeName) that used to just trigger harmless CLI warnings will now cause fatal container crashes because those features have been stabilized and the flags removed. Keep your command array strictly to the bare minimum required to pass the config file.
Trap #4: Pragmatic Security (The Passwordless Docker Network)
The Symptom: You try to follow security best practices by adding authentication to every component, but the stack becomes a house of cards. Specifically, Bitnami’s ZooKeeper refuses to start, and the OpenTelemetry Collector fails to connect to ClickHouse despite "correct" credentials in the DSN.
The Reality: We often fight two battles here. First, modern Bitnami images (often used for ZooKeeper) are secure by default and will simply exit if no authentication is configured. Second, OpenTelemetry configuration files are notoriously finicky with environment variable interpolation in DSN strings—one misplaced character in a password can break the entire pipeline.
The Fix: Embrace network-level isolation as your primary defense. In the official SigNoz approach, and the one we adopted for Verne Software, ClickHouse runs without a password internally. Security is enforced by the fact that the verne_observability network is not accessible from the outside world.
To get the stack running smoothly, apply these two adjustments:
For ZooKeeper: Explicitly allow anonymous logins in your docker-compose.yml. This is acceptable because ZooKeeper is buried deep within your private network.
For ClickHouse: Use a clean, passwordless DSN in your OTel and SigNoz API configurations. It eliminates the "interpolation hell" and simplifies the handshake between services.
signoz_zookeeper:
image: bitnami/zookeeper:3.9.2
environment:
- ALLOW_ANONYMOUS_LOGIN=yes # Required for the stack to boot without complex SASL setup
networks:
- verne_observability
By relying on the Docker network boundary, you reduce configuration complexity while maintaining a solid security posture for your internal telemetry.
Conclusion: The Real Cost of Self-Hosting
After navigating these four traps, my docker-compose ps finally shows a beautiful sight: six containers humming in perfect harmony. From the one-shot UDF initializer to the perfectly synchronized telemetry migrator, the stack is now a reliable engine for our observability needs.
However, the main takeaway from this journey is clear: infrastructure is a full-time job. While owning your telemetry data is empowering, maintaining the underlying "plumbing"—the databases, collectors, and ever-shifting configurations—is a massive tax on engineering time.
This experience is exactly what shapes our philosophy at Verne Software. We believe developers should spend their "innovation tokens" on building core business logic, not on taming infrastructure monsters. It is why we are building the Nautilus ecosystem—to provide powerful "as-a-Service" tools that just work.
If you’re interested in seeing how we’re applying these lessons to simplify developer workflows, we’ve just launched the beta version of our platform at vernesoft.com. You can also dive into our growing documentation at docs.vernesoft.com to see what we’re building next.
Stay pragmatic, and may your logs always be searchable!
