The API server

Last lesson was about how you write objects. This one is about where they actually get handled.

There is one door

The API server is the ONLY component in the cluster that anything talks to.

You submit an object to it. A controller reads objects from it. A machine asks it what it should be running. Nothing goes around it, and nothing reads or writes the database behind it directly, not even the components sitting on the same machine.

That’s a bigger deal than it sounds. Every read and every change goes through one point, so anything the cluster wants to enforce, or notice, only has to be built once.

It’s just an HTTP API

There’s nothing exotic about the protocol. Objects are resources with URLs, and the operations are the usual ones:

GET    /api/v1/namespaces/shop/pods
GET    /api/v1/namespaces/shop/pods/web-7d4f
POST   /apis/apps/v1/namespaces/shop/deployments
PATCH  /apis/apps/v1/namespaces/shop/deployments/web
DELETE /apis/apps/v1/namespaces/shop/deployments/web

The path is just the object’s identity spelled out: which API group, which version, which namespace, which kind, which name. Cluster-wide kinds simply have no namespace segment.

kubectl is an HTTP client for this and not much else. Prove it to yourself once:

kubectl get pods -v=8

You get the request it sent, the headers, and the JSON that came back. A kubectl subcommand is really just a convenience over one or two of these calls, and anything you can do with it you can do with curl and a credential.

Six gates

A request that changes something doesn’t go straight into the store. It walks a fixed sequence, and each step asks one question. The order is the interesting part.

Send something to the API server

You apply a Deployment you, from your kubeconfig

The ordinary case, so you can see what all six gates do when none of them objects.
1. Authentication who is this?
  
  Your kubeconfig holds a client certificate. The API server reads a username out of its subject and a set of groups out of the rest, and that is the whole of what it now knows about you.
2. Authorization are they allowed?
  
  One of your groups is bound to a role that allows create on deployments in this namespace. The question asked here is narrow: this user, this verb, this kind, this namespace.
3. Mutating admission should it be changed?
  
  Your file did not mention a rollout strategy, a grace period, or an image pull policy. The stored object has all three. Defaults are applied here, before anything else ever reads the object.
4. Schema validation is it well-formed?
  
  replicas is a number, the container has a name and an image, and no field is one the Deployment schema has never heard of.
5. Validating admission should it be refused?
  
  Nothing in the namespace objects to the object as it now stands. This is the last chance to say no.
6. Storage write it down
  
  Written, and given a version number. The response comes back as 201 Created and not one container has been started.
Six gates, and the only one that touched a machine was the last, which touched a database. Creating an object is where this story ends.
The same command, with an expired certificate an identity the server cannot establish

Same file, same verb, same namespace. Only the credential has changed.
1. Authentication who is this? 401
  
  The certificate is past its notAfter, so the server cannot establish who is asking. It does not read the body, does not consider whether such a user could create Deployments, and does not say what it would have done. There is nobody to answer about.
2. Authorization are they allowed?
  
  Never reached.
3. Mutating admission should it be changed?
  
  Never reached.
4. Schema validation is it well-formed?
  
  Never reached.
5. Validating admission should it be refused?
  
  Never reached.
6. Storage write it down
  
  Never reached.
A 401 is the shortest path through this diagram. Nothing downstream of the first gate ever learns the request happened.
A pod tries to delete a Deployment a pod, using the token mounted inside it

Every pod can reach the API server, and most of them carry a credential. This one has been talked into using it.
1. Authentication who is this?
  
  The token is real and it names an identity belonging to the shop namespace. This gate has done its job correctly: the caller is exactly who the token says.
2. Authorization are they allowed? 403
  
  Nothing grants that identity delete on deployments. Being known and being allowed are two different questions asked by two different gates, and this is why they are separate.
3. Mutating admission should it be changed?
  
  Never reached.
4. Schema validation is it well-formed?
  
  Never reached.
5. Validating admission should it be refused?
  
  Never reached.
6. Storage write it down
  
  Never reached.
A pod holding a valid credential got exactly as far as its permissions allowed, which was nowhere.
A typo: replicas is quoted you, from your kubeconfig

The file says replicas: "three". Everything else about it is fine.
1. Authentication who is this?
  
  Unchanged. Who you are does not depend on what you sent.
2. Authorization are they allowed?
  
  Still allowed to create Deployments here. Permission is checked against the kind and the verb, not against the contents.
3. Mutating admission should it be changed?
  
  Nothing to default that has not been given.
4. Schema validation is it well-formed? 422
  
  replicas is declared as a 32-bit integer and "three" is a string. The request is rejected as a whole; there is no half-written Deployment left behind, and no controller is ever handed a malformed one.
5. Validating admission should it be refused?
  
  Never reached.
6. Storage write it down
  
  Never reached.
Bad objects fail at the door rather than three components later. That is most of what one door buys you.
A pod that never said what it needs you, from your kubeconfig

The pod names an image and nothing else. The namespace has a policy about pods like that.
1. Authentication who is this?
  
  Unchanged.
2. Authorization are they allowed?
  
  Allowed to create pods here.
3. Mutating admission should it be changed?
  
  A policy in this namespace says a container that does not state a CPU request gets one. Your file did not state one. The object that continues down the path now does, and you never wrote that line.
4. Schema validation is it well-formed?
  
  Well-formed, including the field that was just added for you.
5. Validating admission should it be refused?
  
  Judged as it now stands, with the added request counted.
6. Storage write it down
  
  Written. Read it back and you get the object with the extra field in it.
What you submitted and what got stored are not the same document. This is the single most surprising thing about the request path, and it is why reading an object back is worth doing.
A Deployment the namespace cannot afford you, from your kubeconfig

Ten replicas, each asking for a CPU, into a namespace with a budget that is already mostly spent.
1. Authentication who is this?
  
  Unchanged.
2. Authorization are they allowed?
  
  You are allowed to create Deployments. Whether this particular one is a good idea is not a question this gate can ask.
3. Mutating admission should it be changed?
  
  The usual defaults are filled in.
4. Schema validation is it well-formed?
  
  Perfectly well-formed. It is a valid object that happens to be unaffordable.
5. Validating admission should it be refused? 403
  
  The namespace's budget allows eight CPUs and seven are already committed. This gate runs after defaulting precisely so it can count the object that would really be stored, rather than the shorter one you sent.
6. Storage write it down
  
  Never reached.
Allowed and acceptable are different tests. The first is about you, the second is about the object, and the second one cannot run until the object is finished.

Two of those gates can change or refuse your object after you’ve already been allowed to send it. That’s why what comes back out of kubectl get isn’t always the file you put in, and why a request can pass one gate and get rejected three gates later.

What the response means

When the write succeeds you get the stored object back, with two things attached that you didn’t send: a name for anything you left to be generated, and a version number.

metadata:
  name: web
  namespace: shop
  resourceVersion: "41822"
  uid: 6b1c0b1e-3a7a-4b5c-9f0f-2a5f0f2b8e11

The resourceVersion marks when in the cluster’s history this object was last written. It’s got nothing to do with your app’s version.

Underneath it’s one counter that every write in the cluster bumps, so the numbers only go up and a bigger one always means later. The field is still documented as opaque though, and both things are true for a reason: the ordering is what lets the server answer “everything since 41822”, and the opacity is what stops you building on a detail the project wants to be free to change. You can compare two of them for equality, and you can hand one back.

Handing one back is the useful part. Send an update carrying the version you read, and the write only succeeds if nothing has touched the object since:

PUT /apis/apps/v1/namespaces/shop/deployments/web    resourceVersion: "41822"

409 Conflict
Operation cannot be fulfilled on deployments.apps "web": the object has been
modified; please apply your changes to the latest version and try again

So two things editing one object can’t silently overwrite each other. The loser gets told to read the current object and try again. (This is again reinforcing the concept of eventual consistency)

A watch is not a queue

You could just re-read everything on a timer. It would work, and it would be awful. So the API server offers one more verb:

GET /api/v1/namespaces/shop/pods?watch=true&resourceVersion=41822

The connection stays open and the server sends a small entry every time a pod in that namespace is added, modified or deleted, starting from the version you named. That’s a watch, and it’s how everything in the cluster stays current without polling.

It’s very easy to read that as a message queue, and it really isn’t one. There are two differences and both matter:

A queue	A watch
Delivers every message, eventually	Can drop you, and does
You are expected to process each one	You are expected to re-read the object
Missing one loses information	Missing one loses nothing you cannot re-read

A watch can just end. The connection breaks, the server restarts, or the history you asked to resume from got discarded because it was too old. You don’t recover the lost entries, because they aren’t recoverable. You list everything again and carry on from the version that list came back with.

So nothing built on this can treat an entry as an instruction, because then missing an entry would be a missing instruction.

What it doesn’t do

The API server doesn’t start containers. It doesn’t decide which machine anything runs on. It doesn’t notice that you asked for three and have two.

It answers requests and it writes objects down. Everything in the cluster that looks like it’s doing something is a separate program that got its information the same way you would have, by asking this one.

Where this goes next

Two questions are still open: the database behind this API, and the programs on the other side of it that read everything you write.