How It Works: Cluster Log Shipper as a DaemonSet
Logs are essential for almost all programs. Thses data provide valuable insights for application behavior and troubleshooting clues and can even transform into metrics if needed.
Collecting logs for containers on a Kubernetes Worker Node is not much different than a regular VM. This post explains how it’s done.
- How It Works: Cluster Log Shipper as a DaemonSet (you are here!)
- Getting Started with Grafana Loki, Part 1: The Concepts
- Getting Started with Grafana Loki, Part 2: Up and … (WIP)
General log collecting pipeline
Although different log shippers have different functionalities, the concept is the same.
I use fluentbit and Vector mostly and will take these as examples.
Input (or source) is the phase that you decide where and how you would like to collect logs. It can be syslogs, stdin, HTTP, Docker, files, etc. It can even be other instances of fluentbit or vector if you have more complex architecture.
One of the most common use cases is reading from files. Modern log shippers will know which line to start reading, handle file rotation, scan for new files periodically, etc.
You can check fluentbit’s Tail Input and Vector’s kubernetes_logs source (which is like file with appended Kubernetes metadata).
Transform (or “filter” in fluentbit1) is the phase where you filter, enrich, or modify logs. This part is not mandatory. You can have zero or more transform phases; it’s up to you.
When deployed on edge devices, it’s very likely you don’t want to spend computing power on log processing before sending it. There is a pattern where you just ship it and process it afterward.
Check all the interesting things you can do with your log shipper!
This is also self-explanatory. Output is where you send the logs.
You can send to one or more destinations like stdout, CloudWatch Logs, S3, etc.
Just like the input section mentioned, since you can receive logs from other shippers, you can, of course, send to other shippers.
So, how do we collect container logs from…containers?
Before we talk about how to collect container logs of a Pod on a Worker Node, we need to know where logs are stored.
Containers that print logs through
stderr will be written to files by CRI container runtime2.
The logs are stored in
/var/log/pods/.... You might have seen log shipper using
/var/log/containers, which are symbolic links of files in
Now that we know where the logs are. Let’s take a look at how fluentbit’s chart makes use of
And the Vector one:
The log shipper container can then use
hostPath volume to mount the folder we found above. It only reads, so a best practice here is to make it
Log shipper containers can start consuming logs with the input section above.
By combining these parts in your log collecting pipeline, you can achieve many things.
Preventing data loss is the top priority
However, our log shipping journey has just begun. Your pipeline starts working doesn’t mean it will always work like a charm. One of the things that make admin headache is data loss.
If logs are ingested way too fast (e.g., high service traffic), you should use backpressure mechanism to make sure the shipper itself won’t suddenly increase memory to a unreasonable level.
This is usually done by the disk buffer with a
hostPath volume (again) in order to “persist” the data on the Worker Node. This can help log shipper continues to finish the unsent data after a restart (e.g., crash), a rolling update (e.g., deployment), and keep memory usage stable.
You should prevent data loss at all costs. Admins can only live happily ever after when the settings are thoroughly considered (and hope for machines won’t just break).
Check your log shipper’s settings like the following:
- Fluentbit: Buffering & Storage & Backpressure
- Vector: Backpressure
How about log rotation? Should I take care of it?
Before we migrate container runtime from
containerd because of the Dockershim deprecation, I actually worried a bit since we use Docker’s config to limit the log size.
The answer is,
kubelet does it for you. Check the
containerLogMaxFiles of the Kubelet Configuration (v1beta1) document.
By the time of writing, it defaults to
You can always change the value by Setting Kubelet parameters via a config file.
Can I deploy as a system daemon instead of K8s DaemonSet?
I heard this question before, and of course, you can. These are merely “log files on the Worker Node” as I mentioned above.
However, I don’t see any reason to do it. If you are going to deploy one shipper per Worker Node, you can use Kubernetes’ native DaemonSet. Besides, it allows you to communicate with the API server and “enrich” container logs while making it easier to schedule and update.
- Logging Architecture
- Set Kubelet parameters via a config file
- Kubelet Configuration (v1beta1)
- Volumes: hostPath
It of course does more than “filter”. It can enrich or modify data as well. Check filters for more infomation. ↩︎
See the Container Runtime for more information. ↩︎