Middle of Nowhere

Unfolding the Latency of Unbound's Forwarding Mode

Sun, 18 Jan 2026 11:08:30 +0800

tl;dr: it’s the CNAME chasing behavior.

Lessons Learned When Building My DNS Resolver

Sat, 27 Apr 2024 21:50:39 +0800

It’s simultaneously painful and fun.

Advent of Code 2023 Using Rust

Sat, 02 Dec 2023 16:51:24 +0800

Learning Rust with Advent of Code 2023.

Troubleshooting the Connection Reset Incident

Sat, 23 Sep 2023 15:40:16 +0800

The incident

Recently, a colleague told me one of their applications intermittently throws exceptions saying the connection was reset. The application sends HTTP requests to multiple service providers but the error only happens to a specific one.

Getting Started with Grafana Loki, Part 2: Up and Running

Sat, 15 Jul 2023 22:57:52 +0800

2023/12/29: Added additional learning resources.
2023/12/23: Added paragraph of Prevent OOM using GOMEMLIMIT .
2023/09/08: Added paragraph of Enable route_randomly for Redis cluster .

This post will focus on the day-2 operations of Grafana Loki, providing general suggestions and some caveats when deploying on Kubernetes with AWS.

I highly recommend reading through the Operations section in Grafana’s documentation before diving into this article, as this post serves as a complementary resource.

Additionally, this post will be updated whenever I come across noteworthy information.

Automatically Clean Up Dangling Jobs with Policy Engine

Sat, 09 Jul 2022 21:16:48 +0800

Last year, I was reading about PSP deprecation and started wondering what could be the solutions in the future. Fortunately, there are already several policy engines like OPA Gatekeeper and Kyverno available.

Getting Started with Grafana Loki, Part 1: The Concepts

Mon, 02 May 2022 00:08:45 +0800

2023/07/09: Added the query-scheduler , table manager , and compactor .

Our logging solution for Kubernetes cluster has been CloudWatch Logs for a long time, and we were ok with it. For applications with special requirements, we leveraged S3 for long-term, low-cost storage, then query with Athena .

How It Works: Cluster Log Shipper as a DaemonSet

Fri, 29 Apr 2022 22:38:57 +0800

Logs are essential for almost all programs. These data provide valuable insights for application behavior and troubleshooting clues and can even transform into metrics if needed.

Collecting logs for containers on a Kubernetes Worker Node is not much different than a regular VM. This post explains how it’s done.

Monitor your bandwidth

Sun, 27 Mar 2022 19:57:38 +0800

2022/03/29 update:

In terms of bandwidth, there are actually two metrics called NetworkBandwidthInAllowanceExceeded and NetworkBandwidthOutAllowanceExceeded for ElastiCache like EC2.

They are better metrics to determine whether a node has already reached bandwidth limit. If these values are not yet available or not increasing, it means the node probably either hasn’t exceeded burst bandwidth or burst time.

The Incident

Recently, we saw unexpectedly high traffic during a special event (well, the traffic itself was expected, just didn’t expect this much), and then service went down for a few minutes. The application didn’t show high CPU utilization or memory usage, but API latency was climbing. Checked upstream services, looked all good.

The service team checked the application’s logs and noticed there were many errors related to Redis.

Then checked the Redis’ metrics, CPU is low, memory usage is high, swap is slowly increasing, that looked not good, but shouldn’t cause connection problems. Redis latency is slightly unstable; however, it’s only a few microseconds higher.

What gives?

The Making of Admission Webhooks, Part 2: The Implementation

Sun, 20 Jun 2021 18:21:29 +0800

In part 1 , we briefly went through the concept of admission webhooks. In this post, we are going to build one and deploy it to a cluster.

The Making of Admission Webhooks, Part 1: The Concept

Sun, 02 May 2021 19:44:42 +0800

Recently, we got an internal requirement to send logs to different destinations according to the content. Since our cluster level shipper’s config is already filled with settings to send logs based on the namespace by default, and we would like to make this new feature “pluggable” and can be dynamically set by teams, leveraging admission webhooks appears to be a more reasonable choice. ¹

Query Stub Domains with CoreDNS and NodeLocal DNSCache

Wed, 28 Apr 2021 22:30:42 +0800

Recently, We started migrating our EKS 1.16 clusters to brand new 1.19 ones.

Of all the changes, I am most excited about NodeLocal DNSCache . It not only greatly reduced the time of DNS query latency (mostly because of these search domains ), but issues like conntrack races at very little expense.

Use Default Backend Service on ingress-nginx

Sun, 18 Apr 2021 13:33:52 +0800

We use ingress-nginx as our Ingress Controller .

Due to business requirements, we have lots of domains to handle. It’s unrealistic to create more than one thousand ingress resources, which adds unnecessary loads to control plane and is difficult to maintain and update all of them.

Luckily, ingress-nginx provides a default backend service to handle this situation.

Check TLS Cert Expiry With Node's TLS Module

Fri, 21 Aug 2020 23:25:45 +0800

There are news about some services forget to update TLS certificates from time to time.

It’s even more critical when you are hosting many sites with bunch of domains. Monitoring this is not difficult, it’s often caused by negligence.

Automatically Recover EC2 Instances That Fail Status Checks With CloudWatch Events and Lambda

Sun, 14 Jun 2020 00:23:29 +0800

The Incident

Recently, some of our EKS worker nodes suddenly became unresponsive. When I was checking on the EC2 console, the status check showed “Insufficient Data”.

According to past experience, when underlying hardware somehow got impaired, we will get notifications. However, without much useful information this time, I only did some quick investigation and then had to manually terminate these instances.

If I Had Known This Sooner: $_

Sat, 16 May 2020 17:28:30 +0800

There are always times that I learned something useful and/or cool which I wish I had known sooner. So, I decided to write these little things down. Maybe these kind of things have the potential to be a series.

Prevent AWS CLI V2 From Using Pager

Sun, 23 Feb 2020 15:30:43 +0800

I always like to try new things out. So when I saw AWS CLI v2 is generally available, I just upgraded it without checking changes.

One thing I found different after the upgrade is that, seems all outputs are redirected to things like less. It’s somehow inconvenient when calling AWS CLI in a shell script:

List (Almost) Everything Inside a Namespace of a Kubernetes Cluster

Mon, 17 Feb 2020 22:21:27 +0800

One day, I was preparing to remove a no longer needed namespace of one Kubernetes cluster. Before I did that, I checked what’s inside again to be sure.

So I typed $ kubectl get all and see if I missed something. It turned out that several things were not listed in the output, like the ingresses I was expecting.

Notify Google to Update Sitemap Using Netlify Functions

Sun, 09 Feb 2020 19:07:35 +0800

Currently, this site is hosted on Netlify. I am pretty satisfied and don’t plan to move anytime soon. I also submitted my sitemap to Google for it to index. But the update frequency seems not very high.

Fortunately, Google provides an endpoint for you to notify it. Send a GET request to http://www.google.com/ping?sitemap=${siteMapUrl} and you are done.

But do we have to use curl every time we deploy to tell Google it’s time to fetch our sitemap? Well, life is short, don’t waste time on things like that.

Disable T3 Unlimited Using ASG Lifecycle Hooks, CloudWatch Events, and Lambda

Sat, 08 Feb 2020 17:43:03 +0800

To save costs on testing environments, we use multiple instance types with 100% Spot ratio and lowest-price allocation strategy for several auto scaling groups.

We combined several instance types like c5.xlarge, m5.xlarge, t3.xlarge, t3a.xlarge. It works fine so far, but t3 and t3a instances come with unlimited credits enabled by default. If applications run on these instances suddenly start misbehaving, the cost will increase after accumulated credits burn out.

The Long Way to Windows Container on Amazon EKS: Node Affinity

Sat, 04 Jan 2020 18:59:19 +0800

After dealing with the vpc-resource-controller , I can finally see the IIS page. But a running sample does not mean anything. So I wrapped a few deployment YAML up to see if our workloads work.

The Long Way to Windows Container on Amazon EKS: VPC Resource Controller

Sat, 04 Jan 2020 17:50:35 +0800

Amazon EKS, which was previously called Amazon Elastic Container Service for Kubernetes, is the managed Kubernetes service of theirs. We already have some services (Linux containers) running on production environments for while. It is battle-tested and works well if you ask me.

Due to the characteristic of several Windows-based services ~~that are unlikely to run on other platforms in the foreseeable future~~, we have been discussing the feasibility of running Windows Containers on EKS.

Passed Certified Kubernetes Administrator

Thu, 19 Dec 2019 23:51:35 +0800

I recently passed the Certified Kubernetes Administrator exam. One of the questions I was getting a lot is what motivates me to take the exam in the first place.

Specify DNS Servers Based on Domains on macOS

Fri, 13 Dec 2019 23:56:54 +0800

A few months ago, I noticed that I couldn’t visit to any .dev domains when using the corporation’s network. It was inconvenient, but I didn’t overthink it at that time.

Debug in VSCode: Attach Debugger to Containers

Sat, 28 Sep 2019 14:49:31 +0800

We all need to debug node program from time to time. If you are using VSCode as your IDE, it’s pretty straightforward.

Debugging with local files is even more comfortable. However, when you need to debug node program in a container, there are a few caveats need to notice.

Keep Your Promises with Bluebird

Sun, 21 Jul 2019 15:46:50 +0800

Bluebird is a promise library I’ve used for almost every project. I remember when I was learning promise a few years ago, it took me a while to understand it.

One day, I was searching for some related materials, I found Bluebird. I was wondering what’s the reason for using a third party library when Node.js already has native Promise support at that point? Well, it turns out there are plenty. Although some of its features can be achieved with today’s Node.js, bluebird still can give you a lot of benefits.

Notes: Kinesis Agent for Windows

Sun, 17 Mar 2019 13:45:53 +0800

Recently I have to study Kinesis agent for Windows, so here we are. Actually AWS documents are pretty good for most of the time, this post is just some kind of crash course.

About

Wed, 20 Feb 2019 11:34:25 +0800

I am a dedicated DevOps, SRE, and backend engineer with expertise in building robust and scalable systems. My work involves cloud infrastructure, containers, serverless architecture and implementing best practices for reliability and performance.

One of my passions is sharing my knowledge and insights on DevOps, containers, and programming on my blog, where I delve into practical tips, industry trends, and lessons learned (hopefully not the hard way).

In addition to my technical pursuits, I’m an avid metalhead, finding solace in the energetic and dynamic world of heavy music. When I’m not immersed in coding and automation, you can probably find me exploring new bands.

Passed AWS Certified Developer - Associate

Tue, 19 Feb 2019 19:39:40 +0800

I quit my job before Chinese New Year. Days after that are just plain boring and anxious. It’s basically endless interviews and waiting for the results. I gotta say life like that is not really productive.

It became even more boring when the Chinese New Year vacation started. Almost entire country is not working, let alone interviewing. In order to kill some time, I decided to prepare for AWS certifications. It’s funny because the last time I saw someone having AWS certifications is on one of the interviewers’ blog. I wasn’t sure whether it’s useful or not. But it wouldn’t hurt to get one.

Started a Blog Again

Tue, 19 Feb 2019 15:34:12 +0800

I can’t remember how many times I started a blog. It’s just like new year resolutions, it sounds good and you are so motivated at first, until the passion is gone (which is way faster than you think).

Anyway, here I am, started another blog. This blog will mainly focus on technical things (if these are technical enough 🤷‍♂️) and some life thoughts, and some notes/reflections while I am learning new things.