Use Default Backend Service on ingress-nginx

We use ingress-nginx as our Ingress Controller.

Due to business requirements, we have lots of domains to handle. It’s unrealistic to create more than one thousand ingress resources, which adds unnecessary loads to control plane and is difficult to maintain and update all of them.

Luckily, ingress-nginx provides a default backend service to handle this situation.

From the Command line arguments, we can find a --default-backend-service we can leverage to address this problem.

In our current setup, it’s an older version of chart. It provided an option called controller.defaultBackendService to let user specify the service when host is unknown. Unfortunately, this option was removed.

The newer versions only have controller.defaultBackend.enbabled remained, which is not what we are looking for.

After some searching, it turns out there is still a way to achieve this. As #6430 and #6336 mentioned, a simple extraArgs will do.

So, in the previous version’s

1

controller.defaultBackendService: <namespace>/<service>

can be changed to:

1

controller.extraArgs.default-backend-service: <namespace>/<service>

There are a few caveats:

1. controller.defaultBackend.enbabled should be false.
2. If <namespace>/<service> doesn’t exist, the ingress controller will crash on launch. Which means all other services will not be reachable via ingress controller.
3. If you have lots of domains that can’t be redirected to the same service, this is not the option for you.

Ingress Controller is very important to our system. There was an incident that some ClusterRole resources of it were removed under an unusual circumstance. To be honest, all we had to do is re-install it. However, under the emergency situation like that, we didn’t choose the right version and, sadly, we discovered that default backend service didn’t work. We had to re-install it all over again with an older version.

Since then, we haven’t got time to take a good look into this and hold the impression of “newer version looks like no longer support default backend service” until now.

Keep tracking on the dependencies like libraries, frameworks and services is essential but also somehow tiring. As an SRE/Infra engineer/Developer, there are a lot of things going on.

There are Kubernetes releases you need to follow to see if it brings breaking changes, and if you are running on EKS, there might be defects introduced in certain version of AMI.

I also stumbled into a problem where AWS’s IMDSv2 mechanism caused AWS SDK become slow at the first time it makes request. Let alone other libraries we depend on.

After dealing with all these problems, the changes they made do have a point. It just feels like if I lost track on important projects, I may be doomed someday.

So, enable “watch releases” on the GitHub projects is a must for me now. I have a bunch of RSS feeds to read every day, added a few notifications about new releases is not much for me.

I just don’t know if one day I am not responsible for current projects, how can I ask other people to do the same?