Yewolf
Kubernetes, one idea at a time
Lesson 6 / 19 • 13 min • fundamentals, architecture, storage

etcd, the cluster's memory

One database holds everything the cluster believes. Everything else can be thrown away and rebuilt.

The API server answers questions. It doesn’t remember anything.

Behind it is etcd, and that’s where the cluster’s whole memory lives: every object you’ve written, every object anything else has written, and every report of what’s currently happening.

It’s a key-value store

Not a relational database, and nothing Kubernetes-specific. etcd is a general key-value store with ordering guarantees, and Kubernetes uses it pretty much the way you’d use any of them: object paths as keys, serialized objects as values.

Here’s what’s actually in there after you’ve applied a single Deployment.

The store, opened up

revision 42 003

  • /registry/deployments/shop/web Deployment 

    spec:
  replicas: 3
  strategy:
    type: RollingUpdate
  template:
    spec:
      containers:
        - name: api
          image: web:1.4
          imagePullPolicy: IfNotPresent
status:
  replicas: 3
  readyReplicas: 3

    The object you submitted, plus everything admission added on the way in. There is no second copy anywhere: this record is the cluster's entire memory of your intent.

  • /registry/services/specs/shop/web Service 

    spec:
  clusterIP: 10.96.43.17
  selector:
    app: web
  ports:
    - port: 80

    Written before the Deployment and untouched since, which is why its revision is lower. A revision is just the store's clock at the moment of the write.

  • /registry/replicasets/shop/web-6c8f4b ReplicaSet 

    metadata:
  ownerReferences:
    - kind: Deployment
      name: web
spec:
  replicas: 3

    You did not write this. Something read your Deployment and wrote this one revision later, and it recorded which object it did that for. That link is what makes deleting the Deployment take this with it.

  • /registry/pods/shop/web-6c8f4b-2xk9 Pod 

    spec:
  nodeName: node-2
  containers:
    - name: api
      image: web:1.4
status:
  phase: Running
  podIP: 10.244.2.31

    Nor this. Note that its status lives here too: what is actually happening on a machine is written back into the same store as what was asked for, by whatever is watching that machine.

  • /registry/secrets/shop/web-db Secret 

    data:
  password: c3VwZXJzZWNyZXQ=

    That is base64, and base64 is not encryption. Anything that can read this store can read every password in the cluster, which is why a backup of it is as sensitive as the cluster itself.

  • /registry/minions/node-2 Node 

    status:
  capacity:
    cpu: "4"
    memory: 16Gi
  conditions:
    - type: Ready
      status: "True"

    Nodes are filed under their pre-rename name, which nobody has ever fixed. Its revision is near the top because a node keeps reporting in, which is the main reason an idle cluster still moves through revisions all day.

  • /registry/leases/kube-node-lease/node-2 Lease 

    spec:
  holderIdentity: node-2
  renewTime: "2026-04-11T09:31:14.882Z"

    The current head of the store, and it will not be for long. This is a heartbeat: a tiny object rewritten every few seconds so that the absence of a rewrite means something. It exists so that node objects do not have to be.

  • /registry/events/shop/web-6c8f4b-2xk9.17c3f1 Event 

    reason: Scheduled
message: Successfully assigned shop/web-6c8f4b-2xk9 to node-2
count: 1
type: Normal

    An event is an object like any other, kept in the same store, with an expiry on it. Nothing in the cluster reads these to decide anything.

Two of these eight keys were written by a person. Everything else in the cluster's memory was put there by something reacting to what that person wrote, or by a machine saying what it currently sees.

Two things in that list are worth a second look.

There’s no privileged back channel. Everything in there that you didn’t write was written by a program making a request to the API server exactly like you did, and passing the same six gates.

The key layout is an implementation detail of the API server too, and it has changed before. Nothing should ever read these keys directly, and the one time you’ll see them is in the error messages of somebody who tried.

Revisions are a clock

Every write to the store, of any key, bumps a single counter. That counter is the store’s revision. It’s global rather than per-key, and it only ever goes up.

That’s where resourceVersion came from last lesson. An object’s version is the revision it was last written at, which is why a Service you haven’t touched in a week has a much smaller number than a node reporting in every ten seconds.

One counter for the whole store is what lets you ask the interesting question:

"give me everything that has changed since revision 41822"

A per-key version just can’t answer that, because you’d have to ask per key and you don’t know which keys to ask about. A single ordering over every write in the cluster means one number completely describes how much of history you’ve already seen. Watch is built directly on this.

The history doesn’t last forever though. Old revisions just get compacted away, because otherwise the store would grow forever, and a watcher that asks to resume from a compacted revision gets told its version is too old and has to start over with a full list. That costs nothing, because nothing was relying on having seen every entry: a full list gives you the current state, which is all anybody wanted.

Quorum

etcd is normally run as several members that agree with each other. A write is only a write once a majority of them have it.

MembersMajorityCan lose
110
321
431
532

Four members tolerate exactly as many failures as three do, cost more, and make the majority harder to reach. That’s why every etcd cluster you meet has an odd number of members, and why the usual answers are three and five.

The failure mode is the interesting part. Lose the majority and etcd stops accepting writes, so the API server can no longer accept a change, which means:

  • You can’t create, update or delete anything.
  • No controller can record a decision, so nothing new gets repaired.
  • Every container that was already running keeps running, because nothing on a node needed to ask permission to keep doing what it was doing.

That’s the split between deciding and doing, showing up as an outage. A cluster that has lost its memory just freezes. It stops changing, but it keeps serving traffic the whole time.

Everything else is derivable

Of every component in the cluster, this is the only one holding anything you can’t recreate.

Lose the API server and you start another one. Lose the controllers and they read the store and carry on. Lose a whole node and its work gets placed elsewhere. None of them hold anything that matters, because they all keep it here.

Lose this and the cluster has forgotten what it was for. The containers are still running and nothing knows why, or what should replace them when they stop.

So the backup story is short:

etcdctl snapshot save snapshot.db

That file is the cluster. Not your applications’ data, which lives in whatever storage they were given, but everything the cluster believes about itself. Restoring it is how a control plane comes back from the dead.

It’s also a file containing every Secret, and it gets copied to convenient places.

etcd can be told to encrypt at rest. You hand the API server an encryption configuration naming a provider and a key, and it encrypts the values of the resources you list before writing them:

resources:
  - resources: ["secrets"]
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <32 bytes, base64>
      - identity: {}

There are a few conditions though. It’s off unless you turn it on. It only applies to objects written after you turn it on, so existing Secrets stay readable until something rewrites them. And the key lives on the control plane, right next to the thing it protects, so you’re “only” defending against a stolen disk or a leaked snapshot. Anyone who can reach the API server and authenticate against it can still read them.

What not to put in it

etcd is tuned for a modest number of small objects, read constantly and written rarely, and it holds all of them in memory. It isn’t a database for your application and it isn’t a place to keep large blobs. A cluster that gets slow for no obvious reason is often a cluster where something is writing to it way harder than the designers imagined.

The practical limits are real, and smaller than most people guess. As of writing, the defaults are:

LimitDefaultSet by
The whole store2 GiB, with 8 GiB the largest etcd suggests--quota-backend-bytes
Any single request, so any single object1.5 MiB--max-request-bytes
Data in one ConfigMap or Secret1 MiBKubernetes itself, not etcd

Managed control planes often raise the first one, and you usually can’t see or change it. The second is why a large object gets refused with a message about the request being too large rather than anything about your object.

Filling the store matters because of how it fails. etcd raises a no-space alarm and goes read-only, and it stays read-only after you delete things: somebody has to compact the history, defragment the members and clear the alarm by hand. Until they do, you’ve got the frozen cluster from earlier in this lesson, reached a different way.

It doesn’t have to be etcd

Anything that speaks the etcd v3 API and honours those guarantees can take etcd’s place. That’s really not theoretical. kine implements that API on top of SQLite, Postgres, MySQL or NATS, and it’s what k3s runs by default: the whole cluster’s memory is a file on one machine.

You can also split the store instead of replacing it. Events are high-churn and disposable, so stock Kubernetes will happily send them somewhere else entirely:

--etcd-servers-overrides=/events#https://events-etcd:2379

etcd relies on one Raft group, every write funnelled through one leader, and the keyspace held in memory. Past a certain size that’s just not a database you can grow your way out of. Google has published that GKE’s largest clusters don’t run etcd at all any more, using a Spanner-backed store that speaks the same API, which is what let them take a single cluster to 65,000 nodes.

So etcd is the default and it’s what you should assume you’re running. It’s still only a component. What Kubernetes actually depends on is the contract.

Where this goes next

You’ve now got the API and the database. What’s missing is anything that does something, and that’s the entire rest of the system: programs sitting on the other side of the API server, watching what you write, and quietly making it true.