A Free Model Kept Returning Valid JSON With Shifting Types. The C++ Gate Refused the Ambiguous Cases.

The crash-tagging pipeline looked stable. A free model endpoint received a sanitized stack trace and returned a small JSON object. The integration test accepted the first response. The C++ worker parsed it. The label appeared in the triage queue. Six days later, the worker terminated with an unhandled exception. The log pointed at a field that had always been a string. On that call, the field was a JSON number.

Under the same prompt, the endpoint returned several shapes for the same concepts. The severity field alternated between the string "high" and the integer 3. The confidence field moved between the number 0.93, the string "0.93", and null. The affected_function field was usually a string, but sometimes an empty array. The retry flag changed from false to "false".

The original parser used chained type coercion. It assumed the JSON contract because it had never seen a violation. The code looked like this:

CrashLabel parse_unchecked(const json& j) {
  CrashLabel label;
  label.severity = j.at("severity").get<std::string>();
  label.confidence = std::stod(j.at("confidence").get<std::string>());
  label.affected_function = j.at("affected_function").get<std::string>();
  label.retry = j.at("retry").get<bool>();
  return label;
}

When confidence was a JSON number, get<std::string>() threw a type error. When retry was the string "false", the bool extraction threw another type error. Valid JSON still broke the parser.

The team did not add a prompt rule and hope the model would behave. They built a type gate in front of the parser. The gate accepted only the exact shape the C++ struct required. It returned a decision instead of throwing.

enum class GateDecision { Accept, Reject };

struct CrashLabel {
  std::string severity;
  double confidence;
  std::string affected_function;
  bool retry;
};

GateDecision extract_label(const json& j, CrashLabel& out, std::string& reason) {
  if (!j.is_object() ||
      !j.contains("severity") ||
      !j.contains("confidence") ||
      !j.contains("affected_function") ||
      !j.contains("retry")) {
    reason = "missing required field";
    return GateDecision::Reject;
  }

  const json& severity = j.at("severity");
  const json& confidence = j.at("confidence");
  const json& affected = j.at("affected_function");
  const json& retry = j.at("retry");

  if (!severity.is_string()) {
    reason = "severity is not a string";
    return GateDecision::Reject;
  }
  if (!confidence.is_number()) {
    reason = "confidence is not a number";
    return GateDecision::Reject;
  }
  if (!affected.is_string()) {
    reason = "affected_function is not a string";
    return GateDecision::Reject;
  }
  if (!retry.is_boolean()) {
    reason = "retry is not a boolean";
    return GateDecision::Reject;
  }

  const double c = confidence.get<double>();
  if (c < 0.0 || c > 1.0) {
    reason = "confidence out of range";
    return GateDecision::Reject;
  }

  out.severity = severity.get<std::string>();
  out.confidence = c;
  out.affected_function = affected.get<std::string>();
  out.retry = retry.get<bool>();
  return GateDecision::Accept;
}

Static code review would not catch the drift because the input was remote JSON, not a test fixture. The team wrote a small mutation test. It took one valid seed and twisted each known field into the bad shapes they had already observed.

json mutate_seed(const json& seed, std::size_t i) {
  json m = seed;
  switch (i % 5) {
    case 0:
      m["confidence"] = m["confidence"].dump(); // becomes string
      break;
    case 1:
      m["severity"] = 3; // becomes integer
      break;
    case 2:
      m["affected_function"] = nullptr; // becomes null
      break;
    case 3:
      m["retry"] = "false"; // becomes string
      break;
    case 4:
      m["confidence"] = json::array({0.93}); // becomes array
      break;
  }
  return m;
}

bool gate_rejects_known_drift() {
  const json seed = {
    {"severity", "high"},
    {"confidence", 0.93},
    {"affected_function", "main"},
    {"retry", false}
  };

  for (std::size_t i = 0; i < 5; ++i) {
    const json mutated = mutate_seed(seed, i);
    CrashLabel out;
    std::string reason;

    if (extract_label(mutated, out, reason) == GateDecision::Accept) {
      return false;
    }
    if (reason.empty()) {
      return false;
    }
  }
  return true;
}

The gate's decision table stayed small enough to review in a code review:

The team connected the gate to a free model endpoint they used through MonkeyCode's free model access. The validator ran on the free server option, so the type gate did not need a dedicated CI machine. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free server option removed the cost of a separate worker while the team was still deciding whether the endpoint's labels were useful enough to keep.

The gate was not a complete safety mechanism. It answered one question: is the payload the right shape? It did not answer the harder question: is the label correct? The model could return "severity": "low" for a stack that had a null pointer write. The gate would accept it because the value was a string. A separate semantic validator and human review still mattered.

There were other limits. The mutation test only covered drift the team had already seen. A free model could invent a new shape the fuzzer did not enumerate. A JSON type gate also cannot protect against all C++ extraction errors. Numbers like 1e999 or arrays nested inside a string field still require range and content checks. The gate added a deserialize pass and field-by-field checks. For high-frequency telemetry, that latency might be unacceptable.

Teams that already receive strongly typed protobuf from a stable service will not need the gate. Teams with a real-time budget and no fallback path will find it is extra machinery. The gate earns its place only when the producer is unstable and the consumer is C++.

If a C++ worker is calling get<T>() on remote JSON, the first useful step is to enumerate the malformed shapes that already made it crash. Feed those shapes to a gate like this and fail closed. The parser stops being the final line of defense.