Scalable Kubernetes infrastructure for AI platforms