Authenticate to GKE without kubeconfig
Recently I built a small service that periodically connects to all of our Kubernetes clusters in a given Google Cloud Platform project and reports on what is running in each one. As part of building this service, I had to solve a pretty basic problem: how do I connect to GKE cross-cluster (or even cross-project)? A service running in Kubernetes can talk to its own cluster pretty trivially, but going cross-cluster means thinking about authentication. In this article, I’ll share some Go code to allow GKE-hosted services to connect with external clusters using Google Service Account permissions.
At the outset, I figured that what I wanted to do must be possible, since this is exactly how `gcloud container clusters` works. Running a command like:
creates or updates a local ~/.kube/config file. This config file is how kubectl actually connects to GKE clusters using GCP credentials. I just needed to mimic the same pieces in a Go server. Should be pretty easy right?
If you just want the answer, feel free to jump down to the end and copy-paste my final solution! But if you’re interested in how I got there, read on.
Searching the internet was my first task, but unfortunately I couldn’t find exactly what I needed
The first answer I found on the subject suggested, as one possible solution, bundling the gcloud binary with my service, and having my service literally run gcloud container clusters at runtime. I found that idea problematic:
Bundling
gcloudwould add significant complication to the service image build; I’d be going off rails from the streamlined build process we already have in place for Go servicesInvoking
gcloudis ultimately a stateful operation in the container; I wouldn’t be able to write encapsulated code and forget about it, or analyze multiple clusters in parallel without worrying about mutable config state.
The second answer I found suggested producing a valid ~/.kube/config file in advance, bundling it with my service, and using that to connect to various kube clusters. That didn’t work for me either; I wanted my service to be fully dynamic, not something I’d have to rebuild and redeploy whenever we added or removed a cluster. But the article did give me an idea...
Digging deeper into the library code
If all I needed was a valid ~/.kube/config, then ultimately I just needed to see how the information in ~/.kube/config was used in the Go library code, and try to reproduce the same behavior in memory. With that in mind, I went diving into the code in k8s.io/client-go/tools/clientcmd and k8s.io/client-go/tools/clientcmd/api to see what was going on under the hood. I soon discovered that k8s.io/client-go/tools/clientcmd/api.Config
Is the Go type that represents ~/.kube/config, and clientcmd.NewNonInteractiveClientConfig. These would be the key to creating a usable kubernetes.Interface to a particular cluster.
Trial and Error
My process for getting there was fairly straightforward, if a bit trial-and-error. My idea was to see if I could produce a valid ~/.kube/config that kubectl would accept and successfully operate with. I figured if I could do that, I’d be most of the way there.
I used a google.golang.org/api/container/v1.Service, calling svc.Projects.Zones.Clusters.List for the GCP project id I was interested in, and using - as the zone. This returned a list of all the k8s clusters in a given project, and blessedly included all of the information I’d need to construct my clientcmd/api.Config. I then iterated on my code, dumping the resulting config to disk and comparing it to my existing ~/.kube/config that gcloud container clusters had produced. Eventually, I was able to produce a minimal ~/.kube/config that kubectl itself could actually use to successfully run:
Constructing the kubernetes.Interface
Once I got the config working for kubectl, I needed to get it working in Go. It wasn’t too hard to stitch together the right construction:
The CurrentContext bit is part of what makes the encapsulation work well, I can specify the active cluster I want per k8s client, but the general config is immutable and can be shared across any number of k8s clients.
Last missing piece: GCP auth plugin
My first attempt failed with this error: no Auth Provider found for name "gcp". Thankfully, this time around the internet gave me exactly what I needed (https://github.com/kubernetes/client-go/issues/242). All I needed to do was add a magic import to wire up the gcp auth provider:
As soon as I added that import, everything suddenly worked.
Full solution
To put it all together, here’s a fully working sample, a simple Go binary you can run even on your local machine. (You just need to gcloud auth login and then try to connect to a GCP project you have access to.)
Sample output:
Conclusion
I hope you find this solution useful! At Fullstory, we always like to go the extra mile to make our services as clean, stateless, and resilient as possible. If you have a passion for building solid backend systems or helping manage production systems, we’re always looking for people like you. :)
