Kubernetes Admission Controllers: Initializers

Initializers are one of the many ways that Kubernetes can be extended beyond its “out of the box” functionality. Initializers run during the admission phase of resource creation and allow for a wide variety of behaviors.

In this demo we will create an initializer that annotates a Deployment spec with a specified value. While this is a trivial example, it lays the fundamental structure for creating an initializer controller. This same pattern can be extended to create complex controllers to customize the resource creation process.

The source code for this demo can be found at here.

To complete this demo you will need a Kubernetes cluster with:

https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#initializers

(for information on how to use vagrant and kubeadm to setup the Kubernetes cluster see this post)

During the initializer phase of admission the field metadata.initializers.pending on the resource is populated with the name of each initializer matching the resource spec.

[initializer1.foo.bar, initializer2.foo.bar, initializer3.foo.bar]

The initializer controller watches events on the kube API and checks metadata.initializers.pending for the resource type it is configured for. When the controller sees that the first entry in metadata.initializer.pending matches its own name the controller will perform its action. Once finished, the controller will update the resource spec to remove its name from metadata.initializer.pending. When metadata.initializers.pending is empty the resource is considered initialized.

Our initializer controller will be implemented in Golang and run as a cache.Controller which will watch a set of events and execute functions in response to those events.

https://godoc.org/k8s.io/client-go/tools/cache#Controller

The first step is to create a configuration object to use with the API.

config, err := rest.InClusterConfig()

rest.InClusterConfig() is used when the code will be running from within a Kubernetes Pod. This uses the Pod ServiceAccount and corresponding configuration.

Next is creating the client set object.

clientSet, err := kubernetes.NewForConfig(config)

Client set is a type which allows access to the API clients of various Kubernetes resources. In this example we will be working with the Deployment resource.

The Deployment resource can be found under the appsV1 API.

clientSet.AppsV1()

To configure our controller to watch the correct events we need to create a list watch.

https://godoc.org/k8s.io/client-go/tools/cache#ListWatch

To do that we create a new list watch for the resource type deployments.

restClient := clientSet.AppsV1().RESTClient()
watchlist := cache.NewListWatchFromClient(restClient, ”deployments”, “default”, fields.Everything())

We get a rest client for the appsV1 API then create a list watch for the resource Deployments in the namespace default. We tell the list watch not to filter the results by any field selectors by using fields.Everything().

An unfortunate shortcoming of the NewListWatchFromClient function is that it does not allow us to configure the options for the list and watch requests. This is important because uninitialized resources are not returned from API calls by default. To work around this we do the following:

includeUninitializedWatchlist := &cache.ListWatch{
 ListFunc: func(options metav1.ListOptions) (runtime.Object, error) {
    options.IncludeUninitialized = true
    return watchlist.List(options)
 },
 WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) {
    options.IncludeUninitialized = true
    return watchlist.Watch(options)
 },
}

We create a new list watch object and configure the options for both the list and watch functions. We then re-use the list and watch functions from the list watch object we created using the Deployment API.

Now that we have a correctly configured list watch we can create our controller.

_, controller := cache.NewInformer(includeUninitializedWatchlist, &appsV1.Deployment{}, (30*time.Second),
 cache.ResourceEventHandlerFuncs{
    AddFunc: func(obj interface{}) {
       // TODO: implement code to handle add deployment event
    },
 },
)

We won’t need a store for this controller so we throw away the return value. We pass our list watch which is configured to return uninitialized resources, the type representing a Kubernetes Deployment, and the frequency at which we want the controller to sync with the API (30 seconds). We also pass a single event handler function for the add event.

Now we can implement the code to handle the addition of a new Deployment in our namespace.

func handleAdd(deployment *appsV1.Deployment, clientSet *kubernetes.Clientset) error {}

The handleAdd function should first check whether the Deployment has any initializers.

deployment.ObjectMeta.GetInitializers() != nil

If the Deployment does have some initializers we will need to verify that the name of our initializer controller is first in the list of pending initializers. If it is not the first in the list our function should do nothing and wait until it is first in the list.

pendingInitializers := deployment.ObjectMeta.GetInitializers().Pending
if initializerName == pendingInitializers[0].Name {}

Next we make a copy of the Deployment object and remove our initializer’s name from the list of pending initializers.

initializedDeployment := deployment.DeepCopy()

// Remove self from the list of pending Initializers while preserving ordering.
if len(pendingInitializers) == 1 {
 initializedDeployment.ObjectMeta.Initializers = nil
} else {
 initializedDeployment.ObjectMeta.Initializers.Pending = append(pendingInitializers[:0], pendingInitializers[1:]...)
}

Now we can append the Deployment’s annotations with our custom one.

initializedDeployment.ObjectMeta.Annotations["annotating-initializer"] = "meow"

And finally we marshal the old and updated Deployments, perform a merge, and update the Deployment on the API.

oldData, err := json.Marshal(deployment)
newData, err := json.Marshal(initializedDeployment)
patchBytes, err := strategicpatch.CreateTwoWayMergePatch(oldData, newData, appsV1.Deployment{})
_, err = clientSet.AppsV1().Deployments(deployment.Namespace).Patch(deployment.Name, types.StrategicMergePatchType, patchBytes)

Our controller will be deployed to the Kubernetes cluster; this of course means that we must build a Docker image to run our controller’s code (this is excluded for brevity). We create a standard Deployment spec for our controller with one slight modification; we specify the ServiceAccount the Pod should use.

spec:
   serviceAccountName: demo-init
   containers:

Before we create the Deployment for our controller we will need to create the service account we referenced in our Deployment spec and configure it with the correct RBAC permissions.

Create the service account $ kubectl create serviceaccount demo-init

Create a RBAC role which will allow our controller to initialize, patch, update, watch, and list Deployments in the desired namespace.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: default
name: deployment-initializer
rules:
- apiGroups: ["*"]
  resources: ["deployments"]
  verbs: ["initialize", "patch", "update", "watch", "list"]

Create a role binding between the service account and role.

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: initialize-deployments
 namespace: default
subjects:
- kind: ServiceAccount
  name: demo-init
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: deployment-initializer # this must match the name of the Role
  apiGroup: rbac.authorization.k8s.io

Now our service account will have permission to perform the necessary action to initialize the Deployment and we can create the Deployment for our initializer controller.

Once the initializer controller is successfully created, we will want to create the initializer configuration. This will let the Kubernetes API know which resources should be updated to include our initializers name in metadata.initializer.pending.

apiVersion: admissionregistration.k8s.io/v1alpha1
kind: InitializerConfiguration
metadata:
name: demo-init
initializers:
- name: demo.initializer.cvgw.me
  rules:
    - apiGroups:
        - "*"
      apiVersions:
        - "*"
      resources:
        - deployments

Now we can create a test Deployment to see if our initializer works.

$ kubectl create -f ./kubernetes/sleep.yaml

If everything worked we can now see our annotation on the Deployment spec.

$ kubectl get deployments/sleep -o yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    annotating-initializer: meow
    deployment.kubernetes.io/revision: "1"

A big thank you to Kelsey Hightower whose tutorial was the source for much of this information.