Stateless OODA's in Kubernetes

The fundamental building block in the Kubernetes (k8s) lego box is the concept of a k8s controller, a program that implements a control loop that is responsible for moving k8s towards the desired state.

I thought it was interesting how similar this was to the OODA loop1, which is a decision making process that breaks down into 4 stages:

  1. Observe
  2. Orient
  3. Decide
  4. Act

Now this is relatively banal, most people already take this sort of decision making process into their daily lives. However, what I want to drill down into is how this process is stateless, that is, you don't actually need to persist any information between one iteration loop and into the next.

An example: pretend you are setting up a k8s ReplicaSet. This ReplicaSet is responsible for spinning up a certain number of Docker containers, organized into k8s pods, say 3 pods2. Now k8s handles this behavior like so on the first iteration:

  1. observe the number of pods already running for this ReplicaSet (at iteration 0, this is 0 pods)
  2. compute the difference between the desired amount and current amount (at iteration 0, this is desired-current = 3-0 = 3 pods): https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/replicaset/replica_set.go#L542
  3. decide whether pods need to be spun up or down: https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/replicaset/replica_set.go#L548
  4. spin up pods or spin down pods accordingly

This is exactly an OODA loop3! Kubernetes observes the current state, orients itself, and then decides to act upon the given information. Now, if you pretend one of the 3 pods has failed to spin up. Iteration 1 would look like:

  1. observe the number of pods already running (iteration 1, this is 2 pods)
  2. compute the difference between the desired and current (iteration 1, this is 3-2=1 pod)
  3. decide/act on the information

What's interesting here is that no state was stored between iteration 0 and iteration 1. Iteration 1 did not need any information from iteration 0 to perform its actions, rather, iteration 1 oriented itself based on the current state of the world (that two pods already existed).

Now, you might say that this system is not actually stateless, since there was state encapsulated within the number of pods already running. While this is true, it's important to note that we never explicitly managed the state, that is, we never allocated any variables to track any information generated within the loop. Using this OODA loop, Kubernetes has managed to perform stateful actions while keeping the implementation stateless.

Footnotes:

1

In fact, Michael Gasch's great article on Kubernetes events points this out as well.

2

For simplicity we'll pretend each docker container maps to one pod, although this isn't really necessary information.

3

This example is a little contrived since steps 3 and 4 are merged.

Posted: 2020-12-28
Filed Under: Tech