I haven’t dived too far down the Kubernetes rabbit hole yet, but one thing I was trying to tinker with was deploying Kafka within a k8s cluster. The results were … unsatisfying. The folks at Cockroach Labs have observed similar issues and are offering advice on how to deal with stateful k8s apps.
In short: managing state in Kubernetes is difficult because the system’s dynamism is too chaotic for most databases to handle––especially SQL databases that offer strong consistency.
I’ll note that for Kafka, the odd peccadillos of ZooKeeper configuration make the process “anti-cloud native”. And how to expose long-lived, stateful connections, also seems to work against the Kubernetes grain.
I’m sure someone, somewhere has wrangled through all of these problems but there does seem to be a lot of toil here.