Kubernetes 1.37's kyaml output closes a YAML bug that still deletes data on apply

I wrote a ConfigMap with one field set to ~ and applied it to a real Kubernetes 1.37 API server. I expected an error, the same one I'd just seen for a boolean-looking value. Instead I got configmap/t-tilde created, and the field was gone. Not empty. Gone. kubectl get configmap t-tilde -o json shows a ConfigMap with no data key at all.

That's the bug Kubernetes 1.37 gave kubectl a new tool for. KYAML (KEP-5295) went stable in this release: a stricter, quoted, brace-and-bracket dialect of YAML that kubectl get -o kyaml can print and that kubectl apply -f accepts as ordinary input, because it's still valid YAML underneath. The pitch is that it removes the "Norway problem" — the YAML 1.1 rule that turns an unquoted NO into the boolean false instead of the string "NO", named for the country code. I wanted to know what that bug actually does to a live cluster today, not just what the announcement blog says it does.

Setting up a real 1.37 cluster was the first problem

My plan was kind create cluster --image kindest/node:v1.37.0. It failed at kubeadm's wait-control-plane step. The kubelet inside the node container was crash-looping:

E0925 05:13:26.128098 1015 run.go:72] "command failed" err="failed to validate kubelet
configuration, error: kubelet is configured to not run on a host using cgroup v1.
cgroup v1 support is unsupported and will be removed in a future release"

My sandbox's cgroup hierarchy is v1. I assumed dropping to an older node image would dodge it, so I tried kindest/node:v1.36.1. Same error, same refusal. Kubelet in both 1.36 and 1.37 hard-stops on cgroup v1 hosts — this isn't a 1.37 change, I'd just never hit it before. No kind cluster was happening on this machine at any recent Kubernetes version.

kubectl -o kyaml turned out not to need one. It's client-side formatting, so kubectl create configmap x --dry-run=client -o kyaml works with no server at all. But I wanted to see what happens when a real API server decodes an ambiguous manifest, not just what kubectl prints, so I downloaded the 1.37.0 kube-apiserver and etcd 3.5.17 binaries and ran them directly as plain processes, no kubelet or containerd involved. That's enough for kubectl apply and kubectl get to work against a genuine, current, running Kubernetes API.

What a real apply does with an unquoted Norway-bug value

I wrote thirteen tiny ConfigMap manifests by hand, one ambiguous value each, and applied every one of them to the live 1.37.0 server:

Twelve of thirteen values I'd consider entirely plausible in a real config — a country code, a feature flag, a version string — broke something. Only "hello" survived untouched. But they broke in two completely different ways, and the difference matters more than the headline number.

Eight boolean-looking values and two number-looking values got a loud, immediate rejection:

Error from server (BadRequest): error when creating "flag.yaml": ConfigMap in
version "v1" cannot be handled as a ConfigMap: json: cannot unmarshal bool
into Go struct field ConfigMap.data of type string

That's not silent corruption. ConfigMap.data is typed map[string]string in the Go API, YAML 1.1 parses no as a bool, and the server's decoder refuses to force a bool into a string field. The whole request is rejected before anything is written. Annoying, but safe — you find out immediately, and nothing gets stored wrong.

null and ~ are the ones that don't fit that story. They aren't strings and aren't booleans either, so there's no type mismatch to reject. The decoder just treats a null map value as "nothing here" and drops the key on the way in. No error, no warning, no line in any log I could find. The object gets created; it's just missing a field you asked for.

The rejection doesn't show up in --dry-run=server for the null case

I assumed --dry-run=server in CI would catch all of this before merge. It catches the boolean and number rejections — same BadRequest, no object created. It does not catch the null case:

$ kubectl apply -f dryrun-null.yaml --dry-run=server -o yaml
apiVersion: v1
data:
  plain: hello
kind: ConfigMap
metadata:
  name: t-dryrun-null
  ...

That's a dry run of a manifest whose data block had two keys. The output has one. Dry-run mode reports success and silently shows you the corrupted object — you'd have to already know to check field counts to notice.

Doing it in your own Go code is worse

The apiserver's strict rejection isn't the whole story, though, because the apiserver isn't the only thing decoding YAML in this ecosystem. client-go ships the same sigs.k8s.io/yaml package to any controller or CLI that wants it, and plenty of code calls it directly rather than going through the apiserver's stricter path. I wrote a ten-line Go program that calls yaml.Unmarshal straight into a map[string]string field — the same shape as ConfigMap.data — with the same fourteen values:

type ConfigMap struct {
    Data map[string]string `json:"data"`
}
err := yaml.Unmarshal([]byte(doc), &cm) // err is nil

No error. Eleven of fourteen values changed:

This is the silent-corruption story the KYAML announcement describes, and it's real — it's just not what happens inside the API server itself, which has stricter typed decoding via encoding/json and refuses the mismatch outright. It's what happens in your own code if you use the library the ecosystem hands you the same way I just did. 010 silently becoming the string "8" because it round-tripped through an octal-aware number parser is the kind of thing that survives code review, because nothing about the diff looks wrong.

KYAML actually fixes this, and it's valid input on any kubectl

I generated the same fourteen values as kubectl create configmap norway --dry-run=client -o kyaml ... and fed the output back through the same Go unmarshal call. Zero corrupted, because every value is quoted:

---
{
  data: {
    country: "NO",
    flag: "no",
    zero_padded: "010",
    ...
  },
}

I also hand-wrote a KYAML manifest with a comment and trailing commas — both explicitly disallowed in plain JSON, both allowed here — and applied it to the live server:

---
{
  # comments are fine, JSON doesn't allow this
  data: { note: "trailing commas and comments both allowed", },
}

$ kubectl apply -f commented.kyaml
configmap/norway-commented created

It applied without complaint, on a kubectl that has never heard the word "beta" about this feature and a server with no idea the file wasn't ordinary YAML — because it is ordinary YAML, just a stricter dialect of it. That "works with any kubectl version" claim in the announcement held up exactly as stated.

The cost is size. The same ConfigMap rendered three ways:

+26% over plain YAML for the quoting and braces, still under JSON. For a file a human reads occasionally and a machine reparses constantly, that trade reads as an easy yes.

Concurrency wasn't an interesting angle here — I fired ten identical rejected-value applies at the server in parallel and got ten identical BadRequest errors, no variation. This is a stateless per-request decode, not something that degrades under load.

What I got wrong on the way

I spent the first twenty minutes assuming an older node image would sidestep the cgroup v1 refusal, on the theory that it was a brand-new 1.37 restriction. It isn't — 1.36 refused identically. I'd have found that out in one attempt instead of two if I'd read the kubelet error message properly the first time instead of pattern-matching it to "must be a 1.37 thing" and reaching straight for kindest/node:v1.36.1.

Run it yourself

No cluster needed for the client-side part:

kubectl create configmap norway \
  --from-literal=country=NO --from-literal=flag=no \
  --from-literal=version=1.0 --dry-run=client -o kyaml

For the server-side behaviour, run a bare API server against etcd, no kubelet:

# etcd
etcd --data-dir=etcd-data --listen-client-urls=http://127.0.0.1:2379 \
     --advertise-client-urls=http://127.0.0.1:2379 &

# certs
openssl req -x509 -newkey rsa:2048 -keyout sa.key -out sa.crt -days 1 -nodes -subj "/CN=kube-apiserver"
openssl req -x509 -new -nodes -keyout ca.key -out ca.crt -days 1 -subj "/CN=ca"
echo "local-test-token,admin,admin-uid,system:masters" > token.csv

kube-apiserver --etcd-servers=http://127.0.0.1:2379 \
  --service-cluster-ip-range=10.0.0.0/24 --secure-port=6443 \
  --tls-cert-file=sa.crt --tls-private-key-file=sa.key --client-ca-file=ca.crt \
  --service-account-key-file=sa.crt --service-account-signing-key-file=sa.key \
  --service-account-issuer=https://kubernetes.default.svc \
  --token-auth-file=token.csv --authorization-mode=AlwaysAllow &

kubectl config set-cluster local --server=https://127.0.0.1:6443 --insecure-skip-tls-verify=true
kubectl config set-credentials local-user --token=local-test-token
kubectl config set-context local --cluster=local --user=local-user
kubectl config use-context local

echo 'apiVersion: v1
kind: ConfigMap
metadata: {name: t-flag}
data:
  flag: no' | kubectl apply -f -

And the Go decode check, against sigs.k8s.io/yaml directly:

package main

import (
    "fmt"
    "sigs.k8s.io/yaml"
)

type ConfigMap struct {
    Data map[string]string `json:"data"`
}

const doc = `
data:
  country: NO
  zero_padded: 010
`

func main() {
    var cm ConfigMap
    yaml.Unmarshal([]byte(doc), &cm)
    fmt.Printf("%#v\n", cm.Data) // country:"false" zero_padded:"8"
}

If you maintain manifests that get hand-edited — Helm values files, plain ConfigMaps, anything where a person might type a country code or a version string without thinking about it — grep them for unquoted no, yes, on, off, null, ~, and bare decimal-looking strings before you next touch them, and pipe your generated output through -o kyaml wherever the consumer is something other than kubectl itself. --dry-run=server in CI is worth keeping, but don't trust it to catch a field going missing; it will only tell you about the ones that get rejected outright.