tg-logging-handler: A Complete Implementation Guide

tg-logging-handler is a logging.Handler that delivers Python log records to a Telegram chat through the Bot API. It ships the pieces that always end up hand-rolled in a throwing script: a worker thread that never blocks your code, batching, exponential-backoff retries, 429 rate-limit handling, oversized-message policies, parse_mode escaping, and a bounded queue with counted drops.

Version 0.1.4, MIT license, Python 3.10 and newer. The only runtime dependency is httpx.

Package page: pypi.org/project/tg-logging-handler
source: github.com/0xarchit/tg-logging-handler

Install

pip install tg-logging-handler

Everything below runs offline against mocks unless stated; the live path needs a real bot token.

Setup and environment

Create a bot with @BotFather and find your chat id with @getidsbot. Put the credentials in the environment, which is also how you avoid hardcoding secrets:

export TG_TOKEN=123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11
export TG_CHAT_ID=-100123456789

The handler resolves three things in the same way, argument first, environment second:

The minimal example:

import logging
from tg_logging_handler import TelegramLoggingHandler

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("app")

logger.addHandler(TelegramLoggingHandler(level=logging.INFO))

logger.info("worker started")
logger.error("failed to connect to db: connection refused")

That is the entire integration. A worker thread starts at construction, emit() never blocks or raises, and close() is registered with atexit, so shutdown is covered too. TGLoggingHandler is a shorter alias for the same class.

How it works

Every emit() call copies the LogRecord (records can be mutated or recycled by frameworks, so the handler snapshots them first) and merges the message args immediately. The snapshot goes into a bounded queue.Queue. A single daemon thread, owned by the handler instance, runs the rest of the pipeline:

1. BatchAccumulator.collect() drains the queue until batch_size records or flush_interval elapses.
2. Each record is formatted and escaped for the configured parse_mode.
3. prepare_messages() applies the overflow policy to the joined text.
4. TelegramSender.send_with_retry() POSTs each message to /bot<token>/sendMessage.
5. Outcomes are counted in handler.stats.

Everything network-bound lives on the worker thread. The application thread only ever touches a queue, which is why a slow or dead Bot API cannot stall a request handler. If the worker's formatting or sending ever raises internally, the error goes to stderr (never back through logging, so there is no recursion loop), the batch is counted, and the loop continues.

Batching

The default batch_size=1 sends each record as its own message, which is correct for alerts and wasteful for high-volume services. Batching cuts message count and rate-limit pressure:

handler = TelegramLoggingHandler(
    level=logging.ERROR,
    batch_size=10,       # up to 10 records per message
    flush_interval=10.0, # ...or flush a partial batch after 10 seconds
)

A batch is sent when batch_size records accumulate or when flush_interval seconds pass since the first queued record, whichever comes first. The trade-off is visibility latency: a partial batch sits for up to flush_interval seconds. The flush timer only runs while the queue is non-empty; a quiet handler does not busy-poll.

Retries and rate limits

Transient failures (network errors and 5xx responses) are retried with exponential backoff: 0.5 seconds initially, doubling per attempt, with ±25% jitter, capped at 30 seconds. max_retries=3 is the default, and retries consume that budget.

429 responses are a separate path, because they carry a Retry-After header:

handler = TelegramLoggingHandler(max_retries=5)  # more budget for flaky networks

• The handler sleeps the server-provided Retry-After (clamped to 30 seconds, malformed values fall back to 1 second) and retries, without spending the retry budget.
• Every 429 wait is counted in stats.rate_limited, so a rate-limit storm shows up in the stats even at zero retries (429s never consume the retry budget).
• A non-numeric or infinite Retry-After (some servers send HTTP dates) is normalized instead of crashing the sleep.
• Consecutive 429 waits are capped at 10, so a stuck 429 cannot spin the worker forever. Beyond the cap the batch is dropped, counted as failed, and reported to stderr.
• The first 429 of a sender's lifetime posts a one-time heads-up message into the chat (in the same topic as your logs, see below). It is sent only once, best-effort, and its own failure is swallowed.
• HTTP redirects are never followed: a 302/308 with a Location header (or any non-200 response) surfaces as a permanent failure, so a redirected request can never be mistaken for a delivery.

If Telegram stays rate-limited longer than the cap window, you lose messages. stats.failed is the ground truth for that.

The 4096-character cap and overflow policies

Telegram rejects a sendMessage payload longer than 4096 characters. Tracebacks alone regularly exceed this in a batch, so the handler applies an overflow policy:

handler = TelegramLoggingHandler(overflow="split")     # default
handler = TelegramLoggingHandler(overflow="truncate")  # one message, cut
handler = TelegramLoggingHandler(overflow="drop")      # nothing sent

• split cuts the text into numbered parts, (1/3), (2/3), (3/3), that reconstruct the original exactly when pasted back together. The split is parse_mode-aware: it never breaks a MarkdownV2 escape pair across parts.
• truncate sends a single message cut to the cap with … [truncated] appended. The marker is clamped when the cap is smaller than the marker itself, and nudged off a trailing backslash, so a tiny max_length (and MarkdownV2 escaping) can never produce an oversized payload or a dangling escape pair.
• drop sends nothing; the records are counted as dropped.

The accounting here is deliberate and worth knowing: a drop batch increments dropped by the batch size, while any send path that fails to deliver (including one failed part of a split batch) increments failed by the batch size. A split batch counts as sent only when every part is delivered, never a partial success.

parse_mode escaping

Telegram offers Markdown, MarkdownV2, and HTML formatting, and every parser has characters that will either 400 the request or render wrong when they appear literally in log content. A user-supplied string with an underscore or asterisk is normal in logs. The handler escapes the entire formatted record for the configured mode before anything goes on the wire:

handler = TelegramLoggingHandler(parse_mode="MarkdownV2")

logger.error("version mismatch: v1.2_final vs v1.3_rc1")

v1.2_final arrives as literal text. The truncation marker and split headers are added after escaping, so they render correctly too. With parse_mode=None nothing is escaped.

Forum topics

Groups with topics enabled, and forum supergroups, can split a chat into threads. One handler instance targets exactly one topic, and the topic is configurable per deployment via the TG_TOPIC_ID environment variable instead of code:

export TG_TOPIC_ID=42

handler = TelegramLoggingHandler(topic_id=42)  # same effect as the env var

Implementation details:

• The handler sends the official Bot API message_thread_id field; topic_id is only the user-facing name. The value has to be a positive integer; bool, float, or a string rejected at construction with a ValueError.
• The explicit argument wins over TG_TOPIC_ID. Invalid values (a non-numeric env var, zero or negative ids) fail fast at construction, not at the first send.
• When neither is set, the payload does not carry the field at all, which is byte-identical to older versions: messages go to the group directly, or to the General topic when the group has topics enabled.
• The one-time 429 heads-up notice is posted into the same topic, so an operator watching the topic sees the notice instead of a phantom message in General.
• A wrong or closed topic id comes back as 400 Bad Request: message thread not found. That is a permanent failure: no retries, the batch counts as failed, and it shows in the stats.

The sharp edge: omitting the id does not error, it silently targets General. If you need a specific topic, set the id explicitly.

The bounded queue and backpressure

The queue defaults to 10,000 records. When it is full, queue_full_policy decides:

• block: emit() waits for space. The only policy that can stall your application.
• drop_newest (default): the incoming record is discarded.
• drop_oldest: the oldest queued record is evicted to make room, best-effort under concurrency, with every victim counted.

Every dropped record increments the dropped counter, so losses are visible. For a logging path, drop_newest is usually right: losing the newest line under a flood beats hanging the request handler.

Stats

handler.stats returns an immutable snapshot of seven cumulative counters:

s = handler.stats
print(s.queued, s.sent, s.batches_sent, s.retries, s.rate_limited, s.failed, s.dropped)

• queued: records accepted by emit.
• sent: records Telegram accepted, one batch at a time.
• batches_sent: successful sendMessage rounds.
• retries: transient retries performed.
• rate_limited: 429 Retry-After waits honored (distinct from retries, which 429s never consume).
• failed: records dropped after send failure, or one per batch when formatting raised.
• dropped: records discarded by overflow="drop" or queue-full policies.

The two failure counters do not mean the same thing. dropped is by design (a policy choice), failed is delivery breakdown. The split-batch rule from the overflow section applies: failed parts count the whole batch.

Construction-time validation

Everything that can fail early fails at construction, because a typo discovered at 3am in a log pipeline is expensive:

• Missing or malformed credentials raise TelegramConfigError.
• Env values (TG_TOKEN, TG_CHAT_ID) are stripped of trailing whitespace, so credentials read from .env files or secrets managers with a stray newline validate cleanly.
• batch_size < 1, negative flush_interval, max_retries < 0, queue_maxsize < 0 raise ValueError.
• An unknown queue_full_policy name raises ValueError.
• A non-integer or non-positive topic_id raises ValueError.
• With validate=True (default), a synchronous getMe call verifies the token right away; failures raise TelegramConfigError. The response must be an object with ok == True exactly: a scalar/list JSON body or a merely truthy ok maps to TelegramConfigError too, never a raw AttributeError or a false accept. Set validate=False in tests and offline environments.

These are the only exceptions you can catch, and only during construction: emit() swallows everything per the stdlib contract, and send failures are counted and reported, never raised into your code.

dictConfig

Framework config files work through standard logging.config:

import logging.config

LOGGING = {
    "version": 1,
    "handlers": {
        "telegram": {
            "()": "tg_logging_handler.TelegramLoggingHandler",
            "level": "ERROR",
            "batch_size": 5,
            "topic_id": 42,
            "validate": False,
        },
    },
    "root": {"handlers": ["telegram"], "level": "WARNING"},
}

logging.config.dictConfig(LOGGING)

Shutdown

close() is idempotent and registered with atexit, so it runs at interpreter exit too. It is also race-proof in both directions:

• The shutdown signal is an internal threading.Event, with the queue sentinel as only a wakeup hint; a saturated queue or a drop_oldest eviction can never lose the shutdown and leak the worker thread.
• The sentinel goes onto the queue before the event is set, so the worker can never exit while a sentinel is still about to be enqueued (which would hang any user-side queue.join() forever).
• The worker polls the event between capped waits, so shutdown latency is bounded to about a second even with a partial batch pending and a long flush_interval. The pending batch is still flushed first.
• A shutdown that arrives mid-drain never splits a batch: records that keep flowing still group into one batch, so a burst doesn't flush as N single-record messages.
• close() waits up to shutdown_timeout (default 5 seconds) for the worker; anything still in flight afterwards is finished by the daemon worker in the background, so in a short script call close() explicitly, and read handler.stats after the worker has actually finished, not immediately after close().

Try these next

Small things worth experimenting with, roughly in the order you should try them:

• max_retries=0 to see the failure path fast, then watch stats.failed and the stderr report during an outage.
• batch_size=50 with a fake slow network (the test suite uses respx mocks) and watch batches_sent stay low while sent climbs.
• overflow="drop" on a batch with a giant traceback, then compare dropped to failed in the stats.
• queue_full_policy="block" with a tiny queue_maxsize=2 to feel the backpressure your app inherits, then go back to drop_newest.
• parse_mode="HTML" on a message that contains raw <b> tags from user data.
• A second TG_TOPIC_ID deployment pointed at a different topic of the same group, to split alerts by environment without code changes.
• api_base_url pointed at a self-hosted Bot API server, together with validate=False for a fully offline test rig.
• handler.stats before, during, and after a burst, to see the accounting rules from the overflow section in action.

Where this fits

A Telegram chat is not searchable storage, so use this for alerting and notifications, not log archiving. It coexists fine with an error-tracking service: critical errors page, everything else floods Telegram. Keep the token in the environment, set validate=True in production, and watch failed plus dropped rather than assuming delivery.

The suite behind this package runs offline with mocks, enforces type checking and linting, and gates on coverage; the published wheel ships py.typed markers, so type checkers see the full API.