Controllers

You already know what a controllerControllerA loop that watches one kind of object, compares it against what exists, and closes the gap. is, because you played one earlier. This lesson is about how the real ones are put together, and about the one thing that explains the whole design.

The shape of the loop

Strip a controller down and it’s this:

loop forever:
    take the name of an object off a queue
    read that object, and read what exists because of it
    compare
    if they differ, make one change
    if that failed, put the name back on the queue

That’s the whole thing. Notice what’s not in it: no memory of what it did last time, and no description of what just changed.

It gets handed a name

This is the important bit. A controller’s watch tells it which object is worth looking at. It doesn’t tell it what happened to that object, and the controller doesn’t care.

So the queue holds names. That has consequences that look like small engineering details and really aren’t:

That last row is what makes the system survivable.

There’s a second habit that comes from the same place. Every controller periodically walks every object it looks after, even when nothing has happened at all. If a watch broke, if an entry was missed, if a previous pass quietly failed, the sweep catches it. The correction just arrives late instead of never.

Reconcile has to be safe to run twice

So a controller has to be written so that running it again is harmless. It will run again, on an object it has already handled, with no indication that this is a repeat.

Which means a reconcile pass is never allowed to think in terms of doing a thing. It reads what should exist, reads what does exist, and closes the gap it finds, which is usually no gap at all. Running that a hundred times does the same as running it once, and that property has a name: it’s idempotentIdempotentRunning it again changes nothing beyond what running it once did..

If you’ve ever wondered why Kubernetes has no “create this once” primitive, this is why. Nothing in the system is allowed to need one.

Reading is local, writing isn’t

A controller doesn’t ask the API server every time it wants to look at something. It keeps a local copy of the objects it cares about, kept current by a watch, and reads from that. Otherwise a few dozen controllers in a cluster of any size would flatten the API server between them.

The local copy is very slightly behind, and that’s fine, because writes aren’t local. When a controller does decide to change something it sends a request like anyone else, carrying the resourceVersion it read. If its copy was stale, the write comes back as a conflict.

409 Conflict
Operation cannot be fulfilled on deployments.apps "web": the object has been
modified; please apply your changes to the latest version and try again

At which point the controller doesn’t reconcile the conflict, or merge, or retry the write. It puts the name back on the queue and starts again from a fresh read. A conflict here just tells the loop to go around one more time.

Nobody is in charge

There’s no component that knows how a Deployment becomes a running container. There’s a chain of loops, each of which knows exactly one step and reads only the layer below it.

Run it and watch. Then stop one of the loops and break something, which is the part worth doing.

Each loop only knows its own layer. The deployment controller has never heard of a pod. The ReplicaSet controller has never heard of a Deployment. The scheduler doesn’t know why a pod exists and doesn’t ask.

Stopping one loop doesn’t make another cover for it. Stop the ReplicaSet controller and delete a pod, and the pod stays gone. Nothing else notices, because nothing else was watching for that. Progress on that one thing just stops, and starts again the moment the loop does.

And settled is the normal state. Most of the time every loop looks, finds nothing to do, and goes back to sleep. Work is the exception. A cluster at rest is being re-checked all the time and found correct.

Deleting is a loop too

When the deployment controller creates a ReplicaSet, it stamps it with a note saying which object it made it for. That note is an owner referenceOwner referenceA note on an object saying which object it was created for., and it’s the only link between the two.

metadata:
  name: web-6c8f4b
  ownerReferences:
    - kind: Deployment
      name: web
      uid: 6b1c0b1e-3a7a-4b5c-9f0f-2a5f0f2b8e11

Delete the Deployment and nothing recursively deletes anything. A separate loop, the garbage collector, notices an object whose owner has gone and removes it. Then the pods have no owner either, so it removes those. The cascade runs backwards, at the same speed and by the same mechanism as it ran forwards.

The uid in there is doing real work. Delete a Deployment called web and create a new one with the same name, and the children of the old one don’t get adopted by the new one, because what got recorded was the old object’s identity, not its name.

Where they actually run

Most of the built-in controllers are compiled into one binary, the controller manager, and started as goroutines inside it. That’s just for convenience. They still only talk to each other through objects in the API server.

Run several copies for availability and only one of them is active. They elect a leader by competing to write to a single object, and the others wait. Two controllers reconciling the same objects at once isn’t a disaster, since they’re idempotent, but it’s wasteful and it makes the logs a nightmare.

When two loops want different things

A controller is stubborn by construction. It has one idea of what should be true and it’ll keep writing until reality agrees.

So the failure you eventually meet is two of them with different ideas about the same field. Something scales your Deployment up because it’s busy, your manifest says three, and whichever ran last wins for a few seconds. You get a value that oscillates forever, with both loops working perfectly.

There’s no arbitration for this in the system and there’s no error message. The fix is always to stop one of them caring about that field, and it’s worth knowing about now because the symptom looks like a bug and really isn’t one.

Where this goes next

All of this depends on a controller never needing to be told what happened. What’s left is the stream of things that look exactly like being told what happened, and why nothing in the cluster is allowed to listen.