A Free Model Endpoint Replied Twice, Then Went Silent. The Fix Was a C++ Replay Envelope, Not Retries

Late on a Tuesday, a C++ tooling team noticed their warning classifier was duplicating classifications. The batch runner sent forty compiler diagnostics to a free model endpoint. The first eight came back fine. The ninth returned the same JSON as the fourth. The tenth timed out. The retry loop sent the ninth again. The endpoint answered with a different label. The night ended with a half-written warning database and a confused engineer.

The service was small. A cron job ran clang-tidy on a legacy codebase. It collected warnings. It sent each warning to a free model endpoint with a prompt asking for a severity label: real, noise, or review. The model was supposed to reduce the queue of warnings a human had to inspect. A health check passed before the batch. The team assumed a passing health check meant the endpoint would behave. It did not.

They configured the harness with two settings. One pointed at MonkeyCode's free model access. The other pointed at the free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The failure shapes were different. The first setup returned stale duplicates under load. The second setup dropped connections after a successful warm-up. Neither problem was solved by adding another retry.

The retry loop made both worse. Every retry replayed the same prompt without an idempotency key. The provider could not tell whether a request was a fresh retry or a duplicate batch item. The team had no record of which answers were new and which were copies. A retry also consumed quota. On a free endpoint, a tight retry loop can look like abuse. The team needed a read-side fix that did not depend on provider behavior.

The fix was a replay envelope. Every request was fingerprinted before it left the process. The fingerprint included the prompt, model, temperature, token limit, and schema version. A successful response was stored in a local tape. A failed or malformed response was not stored. On the next failure, the harness could serve the last known good response for that exact fingerprint. The response was marked replayed. Downstream code treated it as a cached artifact, not as a fresh model opinion.

The core data structure is a request key plus a time-bounded tape entry. This is not a cache of arbitrary answers. It is a fallback for an outage. The tape must refuse to match when the prompt or model changes. The tape must expire. The tape must never store truncated JSON. Those rules are cheap to implement in C++ and they turn a flaky endpoint into a bounded recovery path.

#include <chrono>
#include <fstream>
#include <map>
#include <optional>
#include <sstream>
#include <string>
#include <iomanip>

struct RequestKey {
  std::string promptHash;
  std::string model;
  double temperature;
  int maxTokens;
  int schemaVersion;

  std::string fingerprint() const {
    std::ostringstream oss;
    oss << promptHash << '|' << model << '|'
        << std::fixed << std::setprecision(6) << temperature << '|'
        << maxTokens << '|' << schemaVersion;
    return sha256(oss.str());
  }
};

struct ReplayEntry {
  std::string body;
  std::chrono::system_clock::time_point recordedAt;
};

class ReplayTape {
 public:
  void load(const std::string& path);
  void save() const;

  bool get(const std::string& fingerprint, int ttlSeconds,
           std::string& out) const {
    auto it = entries_.find(fingerprint);
    if (it == entries_.end()) return false;

    auto now = std::chrono::system_clock::now();
    auto age = std::chrono::duration_cast<std::chrono::seconds>(
        now - it->second.recordedAt).count();
    if (age > ttlSeconds) return false;

    out = it->second.body;
    return true;
  }

  void put(const std::string& fingerprint, const std::string& body) {
    entries_[fingerprint] = ReplayEntry{
        body, std::chrono::system_clock::now()};
  }

 private:
  std::map<std::string, ReplayEntry> entries_;
  std::string path_;
};

The sha256 helper is standard. It accepts a string and returns a hex digest. The important part is not the hash function. It is the order of operations in the call path.

std::optional<std::string> classifyWithReplay(
    const RequestKey& key,
    ReplayTape& tape,
    const std::string& endpoint)
{
  const int kTimeoutMs = 8000;
  const int kTtlSeconds = 300;

  auto fp = key.fingerprint();
  std::string body;
  int status = postJson(endpoint, encodeRequest(key), body, kTimeoutMs);

  if (status == 200 && validEnvelope(body, fp)) {
    tape.put(fp, body);
    tape.save();
    return body;
  }

  std::string cached;
  if (tape.get(fp, kTtlSeconds, cached)) {
    return markReplayed(cached);
  }

  return std::nullopt;
}

bool validEnvelope(const std::string& body, const std::string& fp) {
  auto env = parseJson(body);
  if (!env) return false;
  if (env->schemaVersion != kCurrentSchema) return false;
  if (env->inputFingerprint != fp) return false;
  if (env->labels.empty()) return false;
  return true;
}

The excerpts omit the JSON parser and HTTP transport. They are not the point. The order is the point: validate, store only on success, replay only on failure.

The test plan made the rules explicit.

The envelope worked because each rule removed a failure mode. The fingerprint prevented prompt A from receiving prompt B's answer. Success-only storage prevented poison entries. The TTL limited staleness. The replayed marker gave downstream code an explicit signal. The team later repeated the same run with the free server option. The failure shape changed from duplicate responses to connection resets. The tape served previously stored responses during the reset window.

Replay is not provider health. It can mask a model that changed under the same fingerprint. For classification, use temperature 0 or an explicit seed when available. For open-ended generation, do not replay. If a provider silently ignores the seed, a replayed answer may look legitimate while being stale. Keep the TTL short and monitor the replay rate. Quota is still consumed on original attempts. Replay only prevents extra retries. The tape does not make a free endpoint private, secure, or compliant. Do not send secrets through it.

Who should not use this approach? Teams that need fresh model output for every call. Teams that send regulated data or need a complete audit trail where replayed artifacts would be misleading. One-off scripts that only need a simple timeout. In those cases, a replay envelope adds bookkeeping without removing the real risk.

The next week, the team logged the replay rate instead of the retry count. A smoke test only proved that the endpoint could answer once. The replay envelope showed how often the batch path actually needed the fallback. That number was the real endpoint health metric.