Recently I was given access to a Kubernetes cluster. The Kubernetes administrator who provided that access wasn't online and I was curious: what level of access and permissions did I have in the cluster? In attempting to query Kubernetes for the answer, I learned some interesting aspects about client certificate authentication, what that means for ascertaining your Kubernetes identity, and some crucial considerations for production environments.
Accessing a Kubernetes Cluster
Access to a Kubernetes cluster is typically given by providing a kubeconfig file. The kubeconfig doesn't "configure Kubernetes" itself. Instead it contains the access protocol that can be used by a Kubernetes client like the kubectl CLI utility or something else such as the OpenLens IDE. In fact, one kubeconfig file can contain definitions for connecting multiple "contexts" across different clusters. That is, it's a handy one-stop shop to drop all of your Kubernetes connection info.
Besides identifying the physical location of the cluster, another critical purpose of the context in a kubeconfig file is that it identifies who you are to the cluster. Direct kubectl to show the configuration:
$ kubectl config view
And it returns an abbreviated view with sensitive parts redacted:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://server.demo.sas.com:6443
name: k8s-viya
contexts:
- context:
cluster: k8s-viya
user: admin@k8s-viya
name: k8s-viya
current-context: k8s-viya
kind: Config
preferences: {}
users:
- name: admin@k8s-viya
user:
client-certificate-data: DATA+OMITTED
client-key-data: DATA+OMITTED
Now let's break this down a little and have kubectl identify the contexts available and which is in use:
$ kubectl config get-contexts
With results:
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* k8s-viya k8s-viya admin@k8s-viya
In my kubeconfig, there's just one context. Notice there's an asterisk "*" under the Current column for the context that kubectl is currently using to connect to Kubernetes. And we can see under the AuthInfo column that the user is identified as admin@k8s-viya.
[ cue record-scratching sound ]
Hold up - there exists no such "user" as admin@k8s-viya in my cluster. When using client certificate authentication (as shown above), that "user" only exists in the definition of the client certificate. It's basically just a label that helps clarify the purpose of the context. But - to be clear - there's nothing defined in Kubernetes that associates admin@k8s-viya with any authentication or authorization structure.
"Okay," the voice I sometimes hear in my head says, "but there's a third-party plugin for kubectl that can tell me the effective user."
And indeed, there is. Once you install the krew plugin manager and have that install the kubectl-whoami plugin, then it's simple to use:
kubectl whoami
And it says:
system:admin
"Ah-ha!" that voice in my head exclaims, "I knew it! Kubernetes says the user is system:admin".
[ cue louder record-scratching sound ]
Yeahhhhh, no. As it turns out, there's nothing defined in my Kubernetes cluster that refers to system:admin either - as we'll see, it comes from the client certificate in the context definition. Whatever that so-called "user" term is, it's intended to be useful for auditing purposes, but as we saw earlier, it might not have direct bearing on who you are to Kubernetes or what you can do.
What Can I Do in Kubernetes?
As it turns out, Kubernetes will let you ask it what capabilities you can perform. Are you a fan of the Twenty Questions game? You can play it with Kubernetes by asking it what "resource" you can "verb":
$ kubectl auth can-i --help
Check whether an action is allowed.
VERB is a logical Kubernetes API verb like 'get', 'list', 'watch', 'delete', etc. TYPE is a Kubernetes resource.
Shortcuts and groups will be resolved. NONRESOURCEURL is a partial URL that starts with "/". NAME is the name of a particular Kubernetes resource. This command pairs nicely with impersonation. See --as global flag.
Examples:
# Check to see if I can create pods in any namespace
kubectl auth can-i create pods --all-namespaces
# Check to see if I can list deployments in my current namespace
kubectl auth can-i list deployments.apps
# Check to see if service account "foo" of namespace "dev" can list pods
# in the namespace "prod".
# You must be allowed to use impersonation for the global option "--as".
kubectl auth can-i list pods --as=system:serviceaccount:dev:foo -n prod
# Check to see if I can do everything in my current namespace ("*" means all)
kubectl auth can-i '*' '*'
Of the examples shown, the most useful one to get a definitive answer is the last one which basically asks if you can "do anything" to "any resource". If the response back is "yes", then you've probably won the game because it's a good bet that you're the cluster-admin (you'd need to specify the ‑‑all‑namespaces parameter to be sure).
But if you're not the cluster-admin, you could go on and on and on asking Kubernetes questions trying to ascertain the limits of your access in the environment. Wouldn't you rather have a more straightforward approach?
Kubernetes RBAC
Kubernetes employs a security scheme known as Rules-Based Access Control (RBAC). There's a lot to it, but let's focus on just two things: Roles and RoleBindings (and their Cluster equivalents).
A Role is basically just a set of permissions - the things you can do (i.e., "resources" that you can "verb"). A Role is limited to given namespace whereas a ClusterRole applies to the entire Kubernetes cluster.
A RoleBinding associates a Role with a list of subjects (i.e., users, groups, and/or service accounts). And again, a ClusterRoleBinding performs a similar function, but applied to the entire Kubernetes cluster.
What we want to do is figure out how the identity referenced in the kubeconfig file corresponds to the RBAC controls of the cluster. The (Cluster)RoleBinding is the linchpin that holds it all together. We just need to connect the dots. As shown above, however, the so-called "users" we found referenced by kubectl and in the kubeconfig haven't been helpful so far.
A Closer Look at the Client Certificate
When we asked kubectl to give us a view of the current context configuration earlier, it hid some critical information with DATA-OMITTED. Let's ask to see that now:
$ kubectl config view --raw -o json
And just looking at the interesting part with the "user" definition here:
"users": [
{
"name": "admin@k8s-viya",
"user": {
"client-certificate-data": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRlZ0F3SUJBZ0lJQUpPNXBkbmtmSFV3Q2dZSUtvWkl6ajBFQXdJd0l6RWhNQjhHQTFVRUF3d1kKYXpOekxXTnNhV1Z1ZEMxallVQXhOamcxTlRVek9UY3hNQjRYRFRJek1EVXpNVEUzTWpZeE1Wb1hEVEkwTURVegpNREUzTWpZeE1Wb3dNREVYTUJVR0ExVUVDaE1PYzNsemRHVnRPbTFoYzNSbGNuTXhGVEFUQmdOVkJBTVRESE41CmMzUmxiVHBoWkcxcGJqQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1xxxDlBd0VIQTBJQUJCaldZMGlpaVZyR0NVVkwKZEIvNnlvbm5GZVdkQVBsaVdZbXlCVEpDakFhMHp4cmhIS0tobGxXSk1TaWpZVVdPalJ6dFJ2SHBlcHE0aDJPTwpEZlJ5YzdpalNEQkdNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WSFNVRUREQUtCZ2dyQmdFRkJRY0RBakFmCkJnTlZIU01FR0RBV2dCUzdhdlZiUDVqZnNWOVZMSFFRUlZrbzNLMnNFekFLQmdncWhrak9QUVFEQWdOSUFEQkYKQWlFQXJLNlA4L1VGSDR3SUZMWmxoU21VakxibGVmODZpWXd1RGdaem0rdHBBaTBDSUd0RlBaeW02OUhmREFiQQpLWVVJOFpacEZibkR1R2h4WlFqRkxmbDdrbE5zCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdFkyeHAKWlc1MExXTmhRREUyT0RVMU5UTTVOekV3SGhjTk1qTXdOVE14TVRjeU5qRXhXaGNOTXpNd05USTRNVGN5TmpFeApXakFqTVNFd0h3WURWUVFEREJock0zTXRZMnhwWlc1MExXTmhRREUyT0RVMU5UTTVOekV3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFSa1FhUysxYmZ2N2EwTWJrbVA5SStiUG4xa3cwL2Z0dU9sQTJYa3ROS3oKWU95VjkwMUFodmxyV3JQTXI4akRpZXNYemlpNnAwd2g1amthbnZwZ2NOMVFvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVXUycjFXeitZMzdGZlZTeDBFRVZaCktOeXRyQk13Q2dZSUtvWkl6ajBFQXdJRFNRQXdSZ0loQUlLMFRyOG1rNHF5UDJiaENpMG5xRmtXUzB1c2hDM0YKckhUdk5rU2V0N3VsQWlFQTFhaVd2Y3p1WnR1NEJVcmJpMXdONzB1VWU3cmF2cWtFSjRYL3htR0R2ME09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K",
"client-key-data": "LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSU12RVNxVTU2K0NGS1g3a0hUN01rb1BLQ0tvc052MXFneXltRTNyaTh0N09vQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFR05aalNLS0pXc1lKUlV0MEgvcktpZWNWNVowQStXSlppYklGTWtLTUJyVFBHdUVjb3FHVwpxxxt4S0tOaFJZNk5ITzFHOGVsNm1yaUhZNDROOUhKenVBPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo="
}
}
],
In particular, the client-certificate-data is most useful. As shown above, it's base64 encoded (not encrypted; a useful technique to ensure clean copying across platforms).
If we get the decoded content (using "base64 -d") of the client-certificate-data and then give that to the openssl utility to explain it:
$ kubectl config view --raw -o json | grep client-certificate-data | awk -F"\"" ' { print $4 } ' | base64 -d | openssl x509 -in /dev/stdin -text
# kubectl = return the full context content
# grep = only show the line with "client-certificate-data"
# awk = return the 4th delimited element (the base64 encoded string)
# base64 = decode the string
# openssl = decode the certificate
We get back:
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 41580943552838773 (0x93b9a5d9e47c75)
Signature Algorithm: ecdsa-with-SHA256
Issuer: CN = k3s-client-ca@1685553971
Validity
Not Before: May 31 17:26:11 2023 GMT
Not After : May 30 17:26:11 2024 GMT
Subject: O = system:masters, CN = system:admin
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
pub:
04:18:d6:63:48:a2:89:5a:c6:09:45:4b:74:1f:fa:
ca:89:e7:15:e5:9d:00:d9:62:59:89:b2:05:32:42:
8c:06:b4:cf:1a:e1:1c:d2:a1:96:55:89:31:28:a3:
61:45:8e:8d:1c:ed:46:d1:e9:7a:9a:b8:87:63:8e:
0d:f4:72:73:b8
ASN1 OID: prime256v1
NIST CURVE: P-256
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication
X509v3 Authority Key Identifier:
keyid:BB:6A:F5:5B:3F:98:DF:B1:5F:55:2C:74:10:45:59:28:DC:AD:AC:13
Signature Algorithm: ecdsa-with-SHA256
30:45:02:21:00:ac:ae:8f:f3:f5:05:1f:8c:08:14:b6:65:85:
29:94:8c:b6:e5:79:ff:3a:89:8c:2e:0e:06:73:9b:eb:69:02:
2d:02:20:6b:45:3d:9c:a6:eb:d1:df:0c:06:c0:29:85:08:f1:
96:69:15:b9:c3:b8:68:71:65:08:c5:2d:f9:7b:92:53:6c
Look at the Subject line:
Subject: O = system:masters, CN = system:admin
Now we're getting somewhere. First of all, we can see that the kubectl-whoami plugin found the identity it reported back from the CN (common name) attribute. Now in some clusters, it's possible that could be defined to refer to a user defined in a RoleBinding. But as established earlier, it's not in the cluster I'm using.
Notice that there's also an O (organization, a.k.a., group) defined: system:masters. Let's try looking for that in the cluster:
$ kubectl get clusterrolebindings -o wide | grep system:masters
With the answer:
NAME ROLE AGE USERS GROUPS
cluster-admin ClusterRole/cluster-admin 28d system:masters
We found the ClusterRoleBinding named cluster-admin and it's associated with the ClusterRole with similar name. According to the documentation describing the cluster-admin ClusterRole, this user has full, unlimited `cluster-admin` privileges in the cluster. The Kubernetes administrator must really like me… I can literally do anything I want in this cluster.
Let's double-check to confirm:
$ kubectl auth can-i "*" "*" --all-namespaces
yes
And so here at the end, we finally found the missing piece that Kubernetes is using to identify who it really thinks I am. The context in the kubeconfig given to me by the Kubernetes administrator grants me full, unlimited permissions via the cluster-admin ClusterRole on all resources by associating with the system:masters group.
Why Does This Matter?
First of all, I personally was somewhat surprised at the one-way direction of creating access control using client certificate authentication as well as the level of isolation. Essentially, it's almost as if the Kubernetes administrator waved a magic wand (to create the RBAC controls and associated client certificate), placed them in a puzzle box (the kubeconfig file with multi-encoded attributes), and gave it to me. So yeah, the Kubernetes administrator knows what's what - but it takes some investigation for me to figure it out in reverse.
Secondly, the arbitrary nature of the "users" defined can be perplexing. They might not exist anywhere in the cluster's RBAC scheme, however they will be logged for auditing purposes as appropriate. Still, since the user names can be so casually defined, there's little to guarantee their effectiveness from an auditing perspective.
Third, I learned that the "system:masters" is a special group whose behavior is hard-coded into Kubernetes, bypassing all RBAC rights checks. There's no changing it. So a kubeconfig file that references the system:masters group provides nearly irrevocable cluster-admin privileges. If a kubeconfig file was misplaced, the only way to invalidate it would be to renew the certificates used by the cluster (or wait for the certificate's defined expiration date).
My key takeaways are that 1) client certificate authentication is difficult to properly secure, 2) take care to define an alternative to the cluster-admin RoleBinding (i.e., avoid associating users with system:masters), and 3) consider using an alternative authentication scheme (like OpenID Connect or Active Directory) for production environments.
H/T
Special thanks to my GEL team colleagues, especially @StuartRogers (identifying weaknesses in the client certificate authentication approach) and @DavidStern (reverse-engineering the X.509 token content). Any mistakes in this post are certainly mine, not theirs. 😉
