The Making of Admission Webhooks, Part 2: The Implementation
In part 1, we briefly went through the concept of admission webhooks. In this post, we are going to build one and deploy it to a cluster.
Let’s keep it simple: this webhook adds a throwaway Redis sidecar container when the pod has the following annotations (why use annotations?):
cache.wtcx.dev/port: <user specified prot>(optional)
cache.wtcx.dev/memory: <user specified memory>(optional)
You can find the complete resources in this repo: github.com/wtchangdm/k8s-admission-webhook-example.
To run our admission webhooks, we need a cluster. k3d lets you run a k3s cluster using Docker. It’s also very lightweight and easy to create/teardown for testing purposes.
k3d, run the following command:
When the cluster is up, you can find a single node in it:
Then we got ourselves a disposable cluster. Let’s not spend too much time on it.
Kubernetes specifically asks the admission webhooks’ scheme to be
https. To fulfill this requirement, we need a certificate. Of course, we can manually generate it, but every time we deploy, we will have to paste it to MutatingWebhookConfiguration and ValidatingWebhookConfiguration’s
It becomes inconvenient when we need to wrap our webhook into a simple helm chart or put it into a version control system. There are many ways to automate this step. For example, kyverno will generate it at runtime; there are also people who use helm hooks with other tools, it’s not a bad idea when you have time to do that.
However, since our goal is just to build a webhook, it would be great when we can focus on that.
cainjectorhelps to configure the CA certificates for: Mutating Webhooks, Validating Webhooks, and Conversion Webhooks.
Pretty self-explanatory, isn’t it? With CA Injector, we don’t need to generate the certificate ourselves, and cert-manager will automatically renew our certificate (it can be signed for a long time, though).
When the certificate is generated, it will create a secret containing the TLS key and certificate as well. We can just mount these two files for our web server pods.
All we need is:
1 2 3 4 5 6 7
$ helm repo add jetstack https://charts.jetstack.io $ helm install \ cert-manager jetstack/cert-manager \ --namespace cert-manager \ --create-namespace \ --version v1.4.0 \ --set installCRDs=true
The webhook design
A webhook is essentially an API. Therefore, we will build a simple API server that serves the response that Kubernetes’ Admission controller expects.
The mutating flow is simple. As you can see, we can skip requests that either don’t have the annotations we are looking for (
cache.wtcx.dev/inject: true) or are dry-run requests. Generally speaking, we can skip
DELETE requests in this case as well. However, it’s easier done in MutatingWebhookConfiguration and/or ValidatingWebhookConfiguration, as we can just opt-out
DELETE requests without ever receving it.
The application itself is written in Node.js with Fasitfy. Again, you can find the source code here.
I just registered two routes, one for mutating admission webhook, and one for validating admission webhook:
And for all these routes, they have several pre-handler hooks before them:
These four fastify hooks (not to be confused with admission webhooks) are the condition blocks in the flow chart above. Each one serves as a middleware, so the handler itself won’t be bothered unless necessary.
It guarantees a request reaches the mutating route (
- It’s not a dry-run request.
- It’s a Pod that has annotation
- All related annotations (
cache.wtcx.dev/memory) are valid if there are any.
- It hasn’t been patched before. (Rember a request can be sent mutliple times?)
Review your work, again
The same logic goes to the validating route (
/v1/hook/cache/validate). Why do we need validating route here? It’s because the Guaranteeing the final state of the object is seen rule. It’s used for making sure that the final state is something you expected.
In my example, the validating route directly rejects the request by throwing
400 with error message in the response body (while the HTTP response itself is still
200, see response format).
Because a legit request shouldn’t have reached this part. It is supposed to be returned early by one of the middlewares. However, you can always run a much more detailed check to see what’s going on and why is the request didn’t pass this phase.
JSONPatch and base64 encoding
Since our example is very straightforward, it doesn’t involve update/replace existing resources like containers, volumes, etc.
We already know that only Pod requests that need to be patched can reach the mutating route; all we need here is to create a Redis container with some settings set to the values that annotations specify:
At mutating route, you can see the following lines:
value is the container object we created above. Since the container is not in the Pod (yet), we are going to
add it at path
/spec/containers/-, as in, insert into the end of
However, we can’t just send this JSON array as a part of the response. Admission controller expects a
base64 encoded string of the JSONPatch result above.
Which looks like:
Let’s decode it just to be sure:
Finally, the response with JSONPatch will look like:
Put it all together
I assume you already have a k3d cluster running and cert-manager installed. In that case, let’s try to deploy it:
First, if you don’t have an existing image, we can build it locally then import into cluster:
With the image imported, we can apply all manifests:
Last, we will apply a deployment that runs
redis-cli that connects to a localhost Redis server:
The deployment itself only includes a single container in the PodSpec. But after patched by our mutating admission webhook, it’s now a Pod containing two containers:
Why was there a container restarted?
Most of the time, you will see the restart number to be 1. That’s because
redis-cli launched faster than the
redis-server we injected.
This demonstration shows how to inject a Redis container with resource set on-demand. But again, this is only for testing purpose, I believe a sane person wouldn’t deploy Redis server like this.
- Dynamic Admission Control
- TLS Certificates for Kubernetes Admission Webhooks made easy with Certificator and Helm Hook?
- Managing a TLS Certificate for Kubernetes Admission Webhook
- In-depth introduction to Kubernetes admission webhooks