The Making of Admission Webhooks, Part 1: The Concept

Recently, we got an internal requirement to send logs to different destinations according to the content. Since our cluster level shipper’s config is already filled with settings to send logs based on the namespace by default, and we would like to make this new feature “pluggable”, leveraging admission webhooks seem to be a more reasonable choice. ¹

In this series, we will learn what admission webhooks are, when, and how to use them. And finally, we will make our own admission webhooks.

What are admission webhooks?

source: A Guide to Kubernetes Admission Controllers

As you can see above, every request sent to kube-apiserver will go through several phases. The first ones are the authentication and authorization to see who the client is and whether it can perform such action.

Before the object is persisted in etcd, the request will go to different admission controllers. Admission controllers are plugins that intercept the request and perform modification or verification on the object based on its purpose. For instance, LimitRanger admission controller will guarantee that namespace’s resource limit won’t be exceeded after creating the object; AlwaysPullImages will modify the image pull policy to Always. At the time of writing, there are more than 40 admission controllers.

You might ask, what do these have anything to do with admission webhooks?

Of all these admission controllers, there are two controllers called MutatingAdmissionWebhook and ValidatingAdmissionWebhook. And you guessed it, they will call the HTTP webhooks you register.

• MutatingAdmissionWebhook sits at the mutating phase in the image above. It can allow or reject this request and can additionally mutate the requests.

• ValidatingAdmissionWebhook sits at validating phase instead. The admission controllers of this phase can only “validate” requests and decide to allow or reject them. Objects at this phase are at their final state, which means they won’t change afterward.

Which applications use admission webhooks?

There are actually plenty of apps that use this mechanism, such as:

• Sidecar injection:
  • Istio uses a mutating admission webhook to inject its sidecar.
• Policy Engines: This allows you to write policies to make sure everything in the cluster is compliant.
  • OPA/Gatekeeper
  • Kyverno
  • …
• ingress-nginx also uses validating webhooks to ensure config’s correctness.
• …and many, many others.

TL; DR

When you need to modify or ensure incoming objects to meet certain criteria, you can count on admission webhooks. Most scenarios can be covered by the aforementioned policy engines unless you have some requirements that you need to implement yourself.

Caveats

When building admission webhooks, the following articles cover at least 90% of them (Let’s take time to appreciate good documents). This part is the summary of several caveats I think to deserve more attention. It is highly advised to go through the articles.

• A Guide to Kubernetes Admission Controllers
• Using Admission Controllers
• Dynamic Admission Control

At an over-simplified level, you need two steps to enable your admission webhooks:

1. Launch your webhook server.
2. Create MutatingWebhookConfiguration and/or ValidatingWebhookConfiguration

We will cover the details at later implementation parts and just continue with these two points bear in mind.

Side Effects

Side effects are the changes made to things that are not what you are monitoring. For instance, if you are monitoring the pod’s creation, but instead of only mutating/validating that pod’s request object, you also make changes to other resources (out-of-band data). It can be a ConfigMap, or not a Kubernetes resource like a Route53 record (rather unlikely, though).

If you make side effects, you need to prepare a reconciliation mechanism such as a control loop to ensure the “out-of-band” data is in the correct, latest desired state.

In my log shipper sidecar case, I was thinking about making a ConfigMap to store log shipper config and mount config as a volume to the containers. However, I am too lazy to write a control loop, so I decided to use init container to call APIs to fetch configs then write to emptyDir volume instead. Otherwise, I have one more ConfigMap to worry about whether it will be changed by other teams.²

Idempotence and Reinvocation Policy

Idempotence is one quality you frequently find in Kubernetes. If you are using mutating admission webhooks, it might be invoked several times depends on your Reinvocation Policy, because your mutating admission webhook is unlikely the only one. If you make changes to pods, chances are there are also mutating webhook doing the same.

If other mutating admission webhooks’ changes might affect your logic, be sure to set reinvocationPolicy correctly. However, the reinvocation after other webhooks is not guaranteed, and the reinvocation times are not guaranteed to be one.

In my case, all I care about is the user’s container, and my sidecar/volume are correctly configured; changes made by other mutating admission webhook shouldn’t be a concern for me. Therefore I can just set reinvocationPolicy to Never (default).

However, no matter what’s your reinvocationPolicy, it is advised to make sure your admission webhooks won’t blindly patch objects. For example, if your goal is to add a sidecar named test, do not add it to the same object every time admission webhook is invoked (although containers with the same wouldn’t have worked in the first place).

High Availability and Failure Policy

In the webhook configuration, there is an option named: failurePolicy. When your webhook timed out or some errors occurred, the admission controller will go ahead and use the failurePolicy you set to decide what to do next. There are two options: Ignore and Fail.

Remember the nginx-ingress I mentioned above? By default, it uses Fail to ensure no ingress can be created when it’s down.

It’s your job to decide whether a request can pass if your webhook is down or unable to process the request. Even it doesn’t work as expected, the cluster administrator should still be notified anyway. Check Monitoring admission webhooks.

Labels or Annotations?

When checking how others put their configs, I found most of them are using annotations. For instance: Istio uses annotation like: sidecar.istio.io/inject: "false", and it’s put in pod annotations.

That made me wonder:

• Why use annotations instead of labels? Isn’t there a objectSelector can be used to filter unwanted resources?
• Why not put in high-level controller’s manifest like deployment? If we only monitor deployment changes, wouldn’t it be easier?

It turns out, label’s format is way more strict (check Syntax and character set):

1. It can’t start with underscore (_).
2. It can’t include spaces ( ).

As for why put annotations in pod’s metadata instead of deployment’s? It’s because changing annotation of deployment doesn’t trigger the rollout process.

That doesn’t mean labels are not useful, though. If your admission webhooks are more like “opt-out” style, as in, monitoring all pods in all namespaces. Then you can use namespaceSelector to exclude the pods inside namespaces like kube-system.

Webserver and Certificate

Implement the webhook shouldn’t be difficult but rather a bit time-consuming. I spent most time testing the JSONPatch and idempotence requirement.

If you plan to run admission webhooks in your cluster, it also requires a certificate with your Kubernetes service name in it. We will cover this later as well.

Check out the best practices

There are still several things to notice before you start building. Check official docs for best practices.

Further Readings

• A Guide to Kubernetes Admission Controllers
• Using Admission Controllers
• Dynamic Admission Control
• Monitoring admission webhooks
• Matching requests: objectSelector
• Best practices and warnings

Cover: https://unsplash.com/photos/kyCNGGKCvyw