Our logging solution for Kubernetes cluster has been CloudWatch Logs for a long time, and we were ok with it. For applications with spcial requirements, we leveraged S3 for long-term, low-cost storage, then query with Athena.
However, as more and more services being containerized and moved into Kubernetes cluster, issues start to emerge:
The time from ingestion to ready for search is suboptimal. Take CloudWatch Insights as an example, it takes roughly 2 minutes to find the logs from my experience.
Searching logs at different places is inconvenient and slow for service team members, let alone comparing them.
CloudWatch Logs cost is going to increase significantly. We can put it into S3, but there will be no metric filters.
Colleagues from very different tech stacks need a lot of time to learn different things.