Infra Blog

The latest product updates and news from Infra

Follow Infra on Twitter ›

How to create users in Kubernetes

August 31, 2022·Matt Williams

Users don't exist in Kubernetes. Depending on your experience and where that experience is focused, that statement may be a bit confusing. You might be a developer building out an application deployed to Kubernetes. You have been given a kubeconfig file that lets you 'login' to the platform. But that file doesn't actually connect to a 'user'. Instead, it's a set of permissions in the form of a role that have been applied to a context. And that context is defined by a set of certificates. All of that combined is what we might refer to as a user, but that term is not actually known anywhere in the platform.

So how do we create this set of certificates to create a user? Well, it's not really that hard, but it is a long string of steps you need to follow and we will go through them now.

Note: This blog is based on a video we released about creating users in Kubernetes. You can find the video here:

Create the User Key

The first step is to create the source key that represents our user. This key is created using a tool like openssl but another popular tool to use is cfssl, created by Cloudflare. Some folks think cfssl is easier to use, and it definitely looks easier to script. But for this example we will use openssl. You can also choose to create the key using a number of different algorithms. For this example we will use ED25519.

Assuming we want to output a key to myuserkey.key:

openssl genpkey -out myuserkey.key -algorithm ed25519

In case you wanted to use RSA 4096 instead of ED25519:

openssl genrsa -out myuserkey.key -length 2048

That key on it's own doesn't have any value with respect to Kubernetes. We need to sign it first.

Create the Certificate Signing Request

In order to sign it we need to create a certificate signing request and then submit that request to Kubernetes. To create that request we run:

openssl req -new -key myuserkey.key -out myuserkey.csr -subj "/CN=myuser/O=edit"

Let's take a closer look at that command. First there is req. The req command creates and processes certificate requests. -key lets us specify the key created with the genpkey command. -out specifies the output csr. Finally there is -subj. There is a lot that we can put in here, but Kubernetes just needs a Common Name and an Organization, which is used to define groups. If you don't include a -subj option, you will be prompted for a number of other options, like OU.

Submit the CSR to Kubernetes

OK, so now the CSR has been created. Now we need to submit that to Kubernetes. As with everything in Kubernetes we need to generate a YAML file.

cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: myuser
spec:
  request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlHaE1GVUNBUUF3SWpFUk1BOEdBMVVFQXd3SWRHVnpkSFZ6WlhJeERUQUxCZ05WQkFvTUJHVmthWFF3S2pBRgpCZ01yWlhBRElRRG9hVURHVHdYRXYyRjVBTk9yQW9QV0ZvWGhSbTlGQjFQSkpuVUFaWlJGZ0tBQU1BVUdBeXRsCmNBTkJBUGZUMldaVGU3R1VJcGVHZnhTbC93WFFTZGZPQmRoeFg5Q05CMEZpOEdteDhIaDcvd2hkVEgwUXRxWmEKWGthZ0h6ejVUbXRKWnNKZDQ4QlpCb3BLdFEwPQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K
  signerName: kubernetes.io/kube-apiserver-client
  expirationSeconds: 86400  # one day
  usages:
  - client auth
EOF

And now for the explanation of this command. The cat <<EOF ... EOF is a nice simple way of running a multiline command. Hopefully apiVersion and kind make sense. We specify the name of the CSR in the metadata. The name can be anything, but its nice to keep the name consistent. The biggest part of this command is under spec.request. The content of this is simply the output of the csr file after it has been base64 encoded.

If you just run cat myuser.csr | base64 you will get the csr with a bunch of newlines added. So the command you want is:

cat myuser.csr | base64 | tr -d "\n"

And then paste the output of that to spec.request above. spec.expirationSeconds is the next decision you need to make. This defines how long the signed certificate will be valid before it expires. You want it to be a relatively short amount of time but not too short. What is too short? Well you will need to distribute a unique cert in the form of a kubeconfig file to each of your users. If you have a mechanism to distribute the file so that it can be automatically updated, then keeping this value as small as minutes is ideal. But you will often see values of months or years.

Getting the Approved Certificate From the Cluster

Now that the CSR is submitted to the cluster, we need to approve the request.

kubectl certificate approve myuser

That command was easy. But in order to use the new certificate, we need to download it and put it into a kubeconfig file. The resource in Kubernetes for Certificate Signing Requests is csr. So you can probably figure out the command to use for this:

kubectl get csr/myuser -o yaml

Of course, that gets us the csr as well as all the metadata about the request. We want just the cert, but its stored base64 encoded. So to get it out we can run this command:

kubectl get csr myuser -o jsonpath='{.status.certificate}'| base64 -d > myuser.crt

That was a lot of steps to get to a cert file. If you look around for this process, you will find a lot of tutorials that tell you to just download the certificate and key for the Certificate Authority in the cluster. The problem with taking that approach is that using managed Kubernetes is a very common scenario. And in most cases with managed Kubernetes you don't have access to the controlplane and thus you cannot scp those two files to your machine. But using the cluster to approve the certificate will work whether you are using managed or unmanaged Kubernetes.

Build the Kubeconfig File

There are three commands you can use to build the kubeconfig file. One to use to set the user, another to set the context, and a third to set the cluster. Unfortunately to set the cluster, you need the CA cert, but remember we don't have access to that since we are probably using a managed cluster. But we have just run a bunch of commands against the cluster to generate the signed certificate. So we have a kubeconfig file for the cluster admin and we can leverage that.

Make a copy of your kubeconfig file and delete everything except the one cluster we created a user for. You will end up with something like this:

apiVersion: v1
clusters:
  - cluster:
      certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURKekNDQWcrZ0F3SUJBZ0lDQm5Vd0RRWUpLb1pJaHZjTkFRRUxCUUF3TXpFVk1CTUdBMVVFQ2hNYourKeyWillBeALotLongerThanThisx2SW4rQTk3cGZWUDBGTGdUOVlteitpdzAwOHFXY1ZKdz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
      server: https://af03b3bogus39-483c-b55f-4cserverurle60c.k8s.ondigitalocean.com
    name: k8s

This part is going to be the same regardless of the role you have.

Now lets add the user.

kubectl --kubeconfig myuserconfig config set-credentials myuser --client-key=testuserkey.key --client-certificate=testuserkey.crt --embed-certs=true

The --kubeconfig myuserconfig specifies which kubeconfig file you want to set the credentials in. config says we are building a kubeconfig file. set-credentials myuser means that we are setting the credentials in the file mention for an entity known as myuser. client-key and client-certificate set the corresponding files. Now if that's all you type, you will get a file that simply refers to each of the files. So if you want to share the kubeconfig file with another person, you will need to also share the certificate and key file. Adding the option embed-certs=true will embed them inline making the file easier to share.

Finally run this command to add a context.

kubectl --kubeconfig myuserconfig config set-context myuser --cluster=k8s --user=myuser

set-context sets the name of the context. And cluster and user set each of those items. So what is a context? Well it is simply a binding between the user and the cluster. When you use a tool like the krew plugin kubectl ctx, it's the context that you are choosing.

Now you are done setting up a new user for your cluster. It's not that difficult to do, though there are a lot of commands you need to run. And quite a lot of it is boilerplate, with a few key items updated for each user. You could definitely script it, and you could also use the great script Brendan Burns created and shared on his GitHub. But even with tools like that, you still need to come up with a way to distribute the generated kubeconfig file. And this could be the most challenging part of the process.

Doing Everything in Infra Instead

Create users in the UI

An alternative to this is to use Infra to build out your users and groups, as well as distribute the files. Just sign in, navigate to Users and fill in the field to enter the user's email address. Then navigate to Clusters and choose your cluster. Now select the user you just created from the dropdown and specify a role. And you are done. Actually that's two more steps than what we did without Infra, which would require coming up with another YAML file to define a role and then another to bind the role to the user, then apply the two files.

And that is everything you need to do to create a user in Kubernetes. It is strange that while the 'user' isn't a first class resource in Kubernetes itself, it is recognized in the kubeconfig file. And it's strange that creating a user isn't a single step command in kubectl. But thankfully tools like Infra make what is a complicated multistep process into a something that can be completed in seconds. If you aren't already using Infra, try it out. With our Quickstart you can be up and running in a few short minutes no matter where your Kubernetes cluster lives.


Shell completion and display improvements

August 8, 2022·Matt Williams

Shell Completion

Infra's CLI now includes shell completion. If you have ever used Infra and had to run the infra help command to remind of the options, this feature is for you. Now press the tab key to see a list of all the commands and subcommands. Run infra completion to generate the completion script and it supports bash, zsh, fish, and powershell.

CLI truncating

If you have a lot of users in a group and then you list them out, the list can be long and it can be difficult to read. Infra's CLI will now truncate that list to make that easier. To see all results, the CLI now has a --no-truncate option.

Group member counts

When viewing groups in the Dashboard and CLI, or using the API, you now see the number of users within each group.

The full changelog can be found here.


Groups in the Infra Dashboard, Google Workspace support

July 22, 2022·Matt Williams

hero

Today we released version 0.14.0 of Infra. The headliner features for this release are support for groups in the Dashboard, and integration with Google Workspaces as an Identity Provider. And of course there was also a number of other features and updates.

In this latest release we now are supporting group management in the Infra Dashboard. This means supporting both groups from OIDC integrations as well as locally defined groups.

Groups in the Dashboard

We also now support working with users and groups from your Google Workspace. You can find the documentation for setting this up here. With this Infra now supports users and groups from Okta, Azure AD, and Google Workspace.

Google Workspaces

There were also a number of other features and updates added to Infra in this release. These include preparation for password requirments, better Identity Provider buttons, pagination in the CLI and preparation for pagination in the UI, and improving our support for postgres as a backing database. You can read all the details in the release here.


Identify old clusters and group management

July 8, 2022·Matt Williams

Last Seen

Infra 0.13.6 was just released. New features include managing groups in the CLI, the ability to easily identify older, stale clusters in both the UI and the CLI, and feedback around when the connector keys will expire.

Identity older, stale clusters

As you work with Infra, you may find that you added clusters that no longer exist. Figuring out which clusters haven't been connected to in a while may be awkward to determine, so we now track and display when the connector was last seen by the server, as well as whether it is currently connected. This feature was added to both the UI and the CLI.

Last Seen

Last Seen UI

Groups

When using the CLI, you will notice there is a new Groups command. This will allow you to create and delete local groups, and add and remove users from those groups. This feature does not affect any groups defined in your IdP and are only for local groups.

Infra Groups

Infra-Version header

Before going into those features, its important to note that there is a breaking change if you are directly using the Infra API. The Infra-Version header is now required for all API requests which will help with future migrations. If you are not using the API in your own applications, you don't need to worry about this.

There are also a number of other new features in the UI, CLI, as well as the API. Review the Changelog for all the details here


Azure AD, typeahead roles and namespaces in the Dashboard

June 30, 2022·Matt Williams

Namespaces

We have finished another sprint and thus have a new release...well, we actually have two releases as of June 27. We added groups support for Azure, some new features in the UI like editing your password and namespace access, and some new features in the CLI like JSON output and a prompt to verify the new server certificate. And then the next day we made a release that added my favorite new feature... some improvements around configuring roles in the UI.

One of the biggest new features for a lot of users is group support in Azure. From day one we supported OIDC for authentication. But while OIDC is used by a lot of providers, how they implement users and groups turns out to be different. So although we didn't push the feature we knew it should work with other OIDC providers and some might require some tweaks to their flavor of OIDC. With Azure Active Directory, some of the basic features worked right away, but groups did not.

As of this release, you can setup the OIDC integration with Azure Active Directory and the grants you assign in Infra for users and groups will work as expected. For now this is just when configuring the provider via the CLI, but we will be enabling that in the UI soon. We will also be going through other OIDC providers to verify they all work as expected. If you have a provider you need to use which doesn't seem to be working, let us know so we can look into it.

Azure OIDC

But you don't have to use an OIDC provider. You could use locally created users. If you do, until recently you couldn't change your own password in the UI. Now you can. Just go down to the user menu at the bottom left of the UI and click Settings. Click the Change button to the right of your password. Your password will be changed.

Also in the UI, you can now see namespaces for your clusters. When you click on Clusters on the left menu, click on the name of the cluster to see all of the namespaces in that cluster. Now you can enter a username and a role and that role will be applied for that namespace on the cluster.

When using the CLI, the output is always formatted for human consumption. But sometimes you want to include Infra in a script and that means formatting the output as something easier for a computer to process. We just added a --format flag which can be used in most places in the CLI. For instance, infra users list --format=json will output the users list as a JSON blob. YAML will be added as a format soon.

When logging in to a new server, your OS probably won't already trust the server's TLS certificate. In the past we would display the notice to use --skip-tls-verify. Now it will show you information about the certificate and prompt you to verify the fingerprint. After trusting it, it will maintain that trust until you logout using infra logout --clear.

Local Cert Trust

My favorite feature added in this release is this last one. And its actually a collection of additions in the UI. The first addition is typeahead. Now in any entry field in the UI where you had to type the name of a user or group, it will give you a list of choices and filter that list down as you type. When you add a user or group for a grant, you now see a list of all the roles on your server. This includes all the roles added with the Infra install, as well as any custom roles with the label app.infrahq.com/include-role set to true. I am so excited about both of these.

When you are ready to upgrade, remember there are three pieces. First update the CLI Instructions for each OS can be found here: https://infrahq.com/docs/install/install-infra-cli. Then update the Infra Helm repo: helm repo update infra. Then upgrade the server: helm upgrade infra infrahq/infra and each of the connectors: helm upgrade infra-connector infrahq/infra.


New Kubernetes roles for port-forward, exec and more

June 10, 2022·Matt Williams

Kubernetes Roles in Infra

This week we released version 0.13.3 of Infra which introduces a few new features and resolved a few issues.

New Kubernetes roles

One thing we hear from a lot of users is that they know they can create roles in Kubernetes, but they wish there were a few more basic roles included out of the box. So we added a few default roles. After installing Infra, you will now see roles for logs, exec, and port-forward, in addition to the existing roles of cluster-admin, edit, and view. This should make everyone's lives much easier.

As we begin to support more OIDC providers, it's important for the API to show which provider a user comes from. But this information wasn't available. Now it is and will make using other providers more useful. We also added the ability to see which users are members of any given group in both the API and the CLI.

$ infra groups list
  NAME        USERS
  Everyone    matt@example.com, patrick@example.com,
              fred@example.com
  Test group  patrick@example.com

Finally in the Web UI, we added Providers. As with the CLI, Okta is supported as an OIDC provider. Other providers will be coming soon in both the UI and the CLI.

Find all the updates in the GitHub repo changelog. Look at releases 0.13.2 and 0.13.3.


Foundations of Infra: OIDC

June 6, 2022·Matt Williams

Infra on openid You might have seen those letters as you navigate the web, but what do they mean. Well the first three letters are for OpenID. You may remember this as an early way to login to some of the web apps you used.

openid

But versions 1 and 2 of OpenID didn't get a lot of adoption. Version 3, or OpenID Connect, got rid of the XML and custom message signature schema used and instead leverages OAuth 2 which uses TLS. TLS is already in use on every client and server platform today.

So what is OAuth 2? Well OAuth is an authorization framework. Version 2 of OAuth came out around 2013 and simplified the flows quite a bit (while also causing a little bit of Internet drama). OIDC wraps around that and provides authentication. OAuth on its own doesn't really provide a standard method for a user to login and provide a password or other verification step. OIDC is what enables that authentication to happen.

For Infra, we use OIDC providers to validate that you are who you say you are. There are quite a few steps involved with that and in the description below this video I will call out some great resources if you want to understand it in more detail. And we use OIDC and OAuth 2.0 throughout the solution to make sure you always have the right access for you.

Infra is made of a few key components. There is a CLI as well as a web UI. Then there is a server component that stores information in a database. You are going to have one server. Then for each cluster you want to connect to, there is a 'connector'.

First you use the CLI to login. When you do that, it gives you a choice depending on how infra has been configured. You might login locally or with an OIDC provider such as Okta (you may see providers referred to as Identity Providers or IdPs).

Once you have verified your identity, the IdP sends a code back to the Infra CLI. That code, which is just a long string of alphanumeric characters, is then sent to the Infra server. The server then sends that to the IdP which says that that code was just generated for this specific user. The IdP returns some tokens to the server that verify information about the user.

Now we know that you are who you say you are and we use this information to generate an access key that you can then use to access the infrastructure. During the configuration of Infra, you would have been granted access to a number of clusters. You now should see all the clusters you have access to in your kubeconfig files, even if those files were blank before logging in.

kubeconfig

When you use a tool such as kubectl to gain access to one of those clusters, it will check with the server to ensure you still have access. The server generates a JavaScript Web Token, or JWT, though it's usually pronounced as JOT. That JWT is sent over to the cluster and the connector on the cluster sees that the JWT has been signed by the server so it can trust that you really have access.

Assuming you have the access required to perform the action you ran, you will get the corresponding results. We will revalidate that you still have access every 5 minutes. That 5 minute interval is configurable, but if you increase the frequency, you may run into rate limits set by the IdP.

And that is how OIDC works in the context of Infra. This post is also available as a video on our YouTube channel.


Foundations of Infra: JWT

May 31, 2022·Matt Williams

Infra on jwt

JWT. JSON Web Token. But it's usually pronounced as jot. If you hear someone say something about a jot, it might actually be about a JSON Web Token. What is a JSON Web Token? It's simply a way to share some fact between services in a verifiable way.

The actual RFC for JWT says that it is a compact, url-safe means of representing claims to be transferred between two parties and that the claim can be cryptographically signed. A claim is any 'piece of information' about a subject. URL-Safe just means using any characters that are allowed in any URI. And cryptographically signed means that the payload cannot be modified without invalidating the signature.

Infra on jwt

If you look around the web for information about JWT, you will find a lot of articles and videos that explain how to use them, along with a bunch that say they are terrible and no one should ever use them. The folks who say they are terrible, are mostly focused on one use-case, and two implementation details.

The use case they are against is that of browser to service authentication, and mostly because, by default, the data in a JWT is unencrypted and can't be revoked before the expiration date defined in the token. You can encrypt the data and set short expiration times, but that complicates things and by the time you do all that, there are other solutions such as session tokens that may be more appropriate.

Infra uses JWT for a different purpose and sets a short expiration time. In the video about OIDC, you saw that we use JWT to record the claim of what this user's email address is and what groups they belong to. That on its own does not guarantee access to resources since the destination determines what access that user or group should have at run time.

The cryptographic signature on the claim allows the client and the destination to be confident that the user is who they claim to be and the groups are the groups they do in fact belong to. If the payload was modified, the signature would be then be invalid and the request would be rejected. Let's take a look at one of these tokens. First the raw JWT looks like this.

eyJhbGciOiJFZERTQSIsImtpZCI6IkpPdjJISmxLYjBKTWNfQVRy
R0JLbWFZNmdRTjdlUU5RWXh0ZXlGaHJDTEE9IiwidHlwIjoiSldU
In0.eyJleHAiOjE2NTQ4ODU0NDcsImdyb3VwcyI6bnVsbCwiaWF0
IjoxNjU0ODg1MTQ3LCJuYW1lIjoibWF0dEBpbmZyYWhxLmNvbSI
sIm5iZiI6MTY1NDg4NDg0Nywibm9uY2UiOiJPeFFwTUZTdjRwIn0.
2NSqYsFTa7ILrQljqKOa5Sx51lcPisITkQGtCCSaesf_KJZHotAB
aIARPPvBiu4bQuGPxOOxA8wAYArcf4qiCQ

A JWT is made of three components. First there is the header, then the payload, and finally the signature. And those three components are separated using dots or periods. So lets pull out the middle component. Each of these components is Base64 encoded. That's what ensures the URL safe part that we mentioned before. Base64 encoding is not encryption, just a standard way of encoding the text.

Now that that component is Base64 decoded, we see that the payload is this. (If you would like to try this yourself, visit jwt.io and paste the encoded text above into the Encoded text block.)

{
  "exp": 1654885447,
  "groups": null,
  "iat": 1654885147,
  "name": "matt@infrahq.com",
  "nbf": 1654884847,
  "nonce": "OxQpMFSv4p"
}

The iat value is the time this token was issued. nbf says that this token is not valid before this time. And exp is the time when this token expires. We can translate those Unix Timestamps to more common date and time formats and can see that this token has a lifetime of 5 minutes. There is also a nonce value in there that ensures the request is unique.

So the last two values in the token are the actual claim. Groups and Name. The name is the user who is running the command and the groups lists out the groups this user is a member of.

That information is used by the destination to determine what roles this user should be allowed to assume.

And that is what a JWT is and how they are used in Infra.

If you would like to see a video version of this document, check out this from our YouTube channel.