The Making of Admission Webhooks, Part 2: The Implementation
In part 1, we briefly went through the concept of admission webhooks. In this post, we are going to build one and deploy it to a cluster.
Let’s keep it simple: this webhook adds a throwaway Redis sidecar container when the pod has the following annotations (why use annotations?):
cache.wtcx.dev/port: <user specified prot>(optional)
cache.wtcx.dev/memory: <user specified memory>(optional)
You can find the complete resources in this repo: github.com/wtchangdm/k8s-admission-webhook-example.
To run our admission webhooks, we need a cluster. k3d lets you run a k3s cluster using Docker. It’s also very lightweight and easy to create/teardown for testing purposes.
k3d, run the following command:
When the cluster is up, you can find a single node in it:
Then we got ourselves a disposable cluster. Let’s not spend too much time on it.
Kubernetes specifically asks the admission webhooks’ scheme to be
https. To fulfill this requirement, we need a certificate. Of course, we can manually generate it, but every time we deploy, we will have to paste it to MutatingWebhookConfiguration and ValidatingWebhookConfiguration’s
It becomes inconvenient when we need to wrap our webhook into a simple helm chart or put it into a version control system. There are many ways to automate this step. For example, kyverno will generate it at runtime; there are also people who use helm hooks with other tools, it’s not a bad idea when you have time to do that.
However, since our goal is just to build a webhook, it would be great when we can focus on that.
cainjectorhelps to configure the CA certificates for: Mutating Webhooks, Validating Webhooks, and Conversion Webhooks.
Pretty self-explanatory, isn’t it? With CA Injector, we don’t need to generate the certificate ourselves, and cert-manager will automatically renew our certificate (it can be signed for a long time, though).
When the certificate is generated, it will create a secret containing the TLS key and certificate as well. We can just mount these two files for our web server pods.
All we need is:
1 2 3 4 5 6 7
$ helm repo add jetstack https://charts.jetstack.io $ helm install \ cert-manager jetstack/cert-manager \ --namespace cert-manager \ --create-namespace \ --version v1.4.0 \ --set installCRDs=true
The webhook design
A webhook is essentially an API endpoint. Therefore, we will build a simple API server that serves the response that Kubernetes’ Admission controller expects.
The flow is pretty simple (click to enlarge). As you can see, we can skip requests that either don’t have the annotations we are looking for (
cache.wtcx.dev/inject: true) or are dry-run requests. Generally speaking, we can skip
DELETE requests in this case as well; however, it’s easier done in MutatingWebhookConfiguration and/or ValidatingWebhookConfiguration, as we can just opt-out
The application itself is written in Node.js with Fasitfy. Again, you can find the source code here.
I just registered two routes, one for mutating admission webhook, and one for validating admission webhook:
And for all these routes, they have several pre-handler hooks before them:
These four hooks are the condition blocks in the flow chart above. Each one serves as a middleware, so the handler itself won’t be bothered unless necessary.
It guarantees when a request reaches the mutating route (
- It’s not a dry-run request.
- It’s a Pod that has annotation
- All other related annotations (
cache.wtcx.dev/memory) are valid.
- It hasn’t been patched before. (Rember a request can be sent mutliple times?)
Review your work, again
The same logic goes to the validating route (
/v1/hook/cache/validate). Why do we need validating route here? It’s because the Guaranteeing the final state of the object is seen rule. It’s used for making sure that the final state is something you expected.
In my example, the validating route directly rejects the request by throwing
400 with error message in the response body (while the HTTP response itself is still
200, see response format).
Because a legit request shouldn’t have reached this part. It is supposed to be returned early by one of the middlewares. However, you can always run a much more detailed check to see what’s going on and why is the request didn’t pass this phase.
JSONPatch and base64 encoding
Since our example is very straightforward, it doesn’t involve update/replace existing resources like containers, volumes, etc.
We already know that only Pod requests that need to be patched can reach the mutating route; all we need here is to create a Redis container with some settings set to the values that annotations specify:
At mutating route, you can see the following lines:
value is the container object we created above. Since the container is not in the Pod (yet), we are going to
add it at path
/spec/containers/-, as in, insert into the end of
However, we can’t just send this JSON array as a part of the response. Admission controller expects a
base64 encoded string of the JSONPatch result above.
Which looks like:
Let’s decode it just to be sure:
Finally, the response with JSONPatch will look like:
Put it all together
I assume you already have a k3d cluster running and cert-manager installed. In that case, let’s try to deploy it:
First, if you don’t have an existing image, we can build it locally then import into cluster:
With the image imported, we can apply all manifests:
Last, we will apply a deployment that runs
redis-cli that connects to a localhost Redis server:
The deployment itself only includes a single container. But after patched by our mutating admission webhook, it’s now a Pod containing two containers:
What can be improved?
Most of the time, you will see the restart number to be 1. That’s because
redis-cli launched faster than the
redis-server we injected.
Rather than hoping application itself to be smart enough to retry, it’s important to think whether this is something needs to be done with admission webhook.
This demonstration shows how to inject a Redis container. But if an application really depends on it, maybe it should just leverage external services, or just carry one in the deployment manifest.
- Dynamic Admission Control
- TLS Certificates for Kubernetes Admission Webhooks made easy with Certificator and Helm Hook?
- Managing a TLS Certificate for Kubernetes Admission Webhook
- In-depth introduction to Kubernetes admission webhooks