Last year, I was reading about PSP deprecation and started wondering what could be the solutions in the future. Fortunately, there are already several policy engines like OPA Gatekeeper and Kyverno available.
Our logging solution for Kubernetes cluster has been CloudWatch Logs for a long time, and we were ok with it. For applications with spcial requirements, we leveraged S3 for long-term, low-cost storage, then query with Athena.
However, as more and more services being containerized and moved into Kubernetes cluster, issues start to emerge:
The time from ingestion to ready for search is suboptimal. Take CloudWatch Insights as an example, it takes roughly 2 minutes to find the logs from my experience.
Searching logs at different places is inconvenient and slow for service team members, let alone comparing them.
CloudWatch Logs cost is going to increase significantly. We can put it into S3, but there will be no metric filters.
Colleagues from very different tech stacks need a lot of time to learn different things.
Logs are essential for almost all programs. Thses data provide valuable insights for application behavior and troubleshooting clues and can even transform into metrics if needed.
Collecting logs for containers on a Kubernetes Worker Node is not much different than a regular VM. This post explains how it’s done.
In terms of bandwidth, there are actually two metrics called NetworkBandwidthInAllowanceExceeded and NetworkBandwidthOutAllowanceExceeded for ElastiCache like EC2. It’s a better metric to determine whether a node has already reached bandwidth limit.
If these values are not yet available or not increasing, it means the node probably either hasn’t exceeded burst bandwidth or burst time.
Recently, we saw unexpectedly high traffic during a special event (well, the traffic itself was expected, just didn’t expect this much), and then service went down for a few minutes. The application didn’t show high CPU utilization or memory usage, but API latency was climbing. Checked upstream services, looked all good.
The service team checked the application’s logs and noticed there were many errors related to Redis.
Then checked the Redis’ metrics, CPU is low, memory usage is high, swap is slowly increasing, that looked not good, but shouldn’t cause connection problems. Redis latency is slightly unstable; however, it’s only a few microseconds higher.
What gives?
In part 1, we briefly went through the concept of admission webhooks. In this post, we are going to build one and deploy it to a cluster.
Let’s keep it simple: this webhook adds a throwaway Redis sidecar container when the pod has the following annotations (why use annotations?):
W.T. Changcache.wtcx.dev/inject: truecache.wtcx.dev/port: <user specified prot> (optional)cache.wtcx.dev/memory: <user specified memory> (optional)You can find the complete resources in this repo: github.com/wtchangdm/k8s-admission-webhook-example.
Recently, we got an internal requirement to send logs to different destinations according to the content. Since our cluster level shipper’s config is already filled with settings to send logs based on the namespace by default, and we would like to make this new feature “pluggable”, leveraging admission webhooks seem to be a more reasonable choice. 1
Recently, We started migrating our EKS 1.16 clusters to brand new 1.19 ones.
Of all the changes, I am most excited about NodeLocal DNSCache. It not only greatly reduced the time of DNS query latency (mostly because of these serach domains), but issues like conntrack races at very little expense.
We use ingress-nginx as our Ingress Controller.
Due to business requirements, we have lots of domains to handle. It’s unrealistic to create more than one thousand ingress resources, which adds unnecessary loads to control plane and is difficult to maintain and update all of them.
Luckily, ingress-nginx provides a default backend service to handle this situation.
There are news about some services forget to update TLS certificates from time to time.
It’s even more critical when you are hosting many sites with bunch of domains. Monitoring this is not difficult, it’s often caused by negligence.
Recently, some of our EKS worker nodes suddenly became unresponsive. When I was checking on the EC2 console, the status check showed “Insufficient Data”.
According to past experience, when underlying hardware somehow got impaired, we will get notifications. However, without much useful information this time, I only did some quick investigation and then had to manually terminate these instances.