Managing access to logs for specific SAS Viya deployments with Kibana
System logs reveal what happens behind the scenes in a SAS Viya deployment. But SAS Viya system logs include sensitive data, so administrators need to manage who can access those logs.
When K8s clusters have more than one Viya deployment, administrators may need a way to manage access to logs by deployment, so that a user of one deployment can view logs from that deployment, but not from other deployments.
Three scripts included in the SAS Viya Monitoring for Kubernetes project, and documented in the SAS Viya Administration Guide, create and delete Kibana backend roles, role mappings and users, so that the new users are restricted to seeing logs from a single SAS Viya namespace. These scripts greatly simplify what may otherwise be an unfamiliar and somewhat confusing procedure. In this article we'll explore where to find those scripts, how to run them, and we'll see what they do.
Where to find the logging module administration scripts
If you have not already cloned the SAS Viya Monitoring for Kubernetes project from GitHub, here's how:
git clone https://github.com/sassoftware/viya4-monitoring-kubernetes.git
Cloning this project creates a new subdirectory called viya4-monitoring-kubernetes in the current directory. There is a main README.md file in that subdirectory, and there are further README.md files in the monitoring and logging directories below it. Follow the instructions in the SAS Viya: Administration documentation, or those in the README.md files to deploy and configure the two modules of this project, for Monitoring and Logging. Administration scripts for the logging module of SAS Viya Monitoring for Kubernetes live under the logging/bin subdirectories of that directory:
[cloud-user@pdcesx11147 ~]$ ls -1 viya4-monitoring-kubernetes/logging/bin/
change_internal_password.sh
.
. [omitting several other scripts]
.
security_create_rbac.sh
security_delete_rbac.sh
user.sh
Of these, the last three are the ones we will focus on in this article.
Two Kubernetes Namespaces: a viya_namespace and a logging_namespace
All of the scripts or commands covered in this article require the name of a Kubernetes namespace as a parameter, or in an environment variable, or both. The topic covered by this article involves two sorts of Kubernetes namespace, so to avoid confusion:
- I will refer to a namespace containing a SAS Viya deployment with the placeholder name viya_namespace.
- For example, my Kubernetes cluster has two SAS Viya deployments, named
gelcorpandgelinstitute. So, I would substitute one or other of those namespace names for viya_namespace wherever a command refers to it.
- For example, my Kubernetes cluster has two SAS Viya deployments, named
- I will refer to a namespace containing the SAS Viya Monitoring for Kubernetes project's logging components as the logging_namespace.
- In this same cluster, the logging components of SAS Viya Monitoring for Kubernetes are deployed in a namespace called
ops4viyalog. So, I would substituteops4viyalogfor logging_namespace. You will need to make similar substitutions in your environment if you try this.
- In this same cluster, the logging components of SAS Viya Monitoring for Kubernetes are deployed in a namespace called
Read the documentation and README.md files carefully. They often just use the placeholder 'namespace' without specifying which one they are talking about, because it is nearly always the viya_namespace. The viya_namespace and logging_namespace placeholders are not likely the actual strings you would use in your commands, unless that is what your namespaces are actually called!
How to use the Kibana security Role Based Access Control (RBAC) scripts
As described in the SAS Viya Administration documentation for Logging and Monitoring, under 'Set Up Access for Kibana Users', these scripts are intended to be used as follows, from within the logging/bin subdirectory.
| Script and arguments | Use |
security_create_rbac.sh viya_namespace |
Set up Kibana role-based access for a SAS Viya namespace called viya_namespace. This script creates the search_index_viya_namespace role and the viya_namespace_kibana_users back-end role. |
security_delete_rbac.sh viya_namespace |
Delete the roles and mappings created by security_create_rbac.sh, for a SAS Viya namespace called viya_namespace. |
user.sh create --namespace viya_namespace [--user username] [--password password] |
Create a Kibana user with the viya_namespace_kibana_users backend role, granting it access to Kibana, but only to see logs from the namespace called viya_namespace.
The default value for The default value for --password password is the |
user.sh delete --user username |
Delete a Kibana user that was created by this user.sh script. |
Important: As the documentation explains, these scripts do not validate the viya_namespace value. If no such namespace exists, the scripts will still run quite happily, and set up roles, role mappings and users in Kibana that refer to the nonexistant namespace.
Tip 1: Notice that the security_create_rbac.sh and security_delete_rbac.sh scripts currently take the namespace as an argument, without --namespace before it. The user.sh script has more parameters, and so the namespace parameter must be preceded by --namespace (or -ns).
Tip 2: Run any of the commands with a -h or --help argument to see more detailed usage information. Run user.sh create --help or user.sh delete --help for usage information specific to the user create and delete options.
Tip 3: You should run the scripts as a user who has administrator kubectl access to your logging_namespace (or at least, as a user who has access to get the Kibana admin username and password secrets from your logging_namespace). All the scripts use kubectl commands to get the Kibana administrator username and password, and to set up Kubernetes port-forwarding.
Tip 4: Several of the scripts output a message like this:
INFO You may see a message below about a process being killed; it is expected and can be ignored.
The scripts create a temporary Kubernetes port-forwarding process, to enable them to communicate with the Kibana services in your K8s cluster. When the scripts are done, they kill the port-forwarding process. This is normal.
The LOG_NS environment variable
Most, or all scripts in the logging/bin directory expect an environment variable called LOG_NS to be set. It should contain the logging_namespace name mentioned earlier. I have used logging_namespace as a placeholder for this name in the examples below - replace it with the actual name of your logging namespace! Here are three ways to set this environment variable:
Option 1:
export LOG_NS=logging_namespace
[command]
Option 2:
export LOG_NS=logging_namespace ; [command]
Option 3:
Define a USER_DIR environment variable, referring to a directory containing one or more specific user configuration files for the SAS Viya Monitoring for Kubernetes project.
See https://github.com/sassoftware/viya4-monitoring-kubernetes/blob/master/logging/README.md#customize-t... for a detailed description of the configuration files for the logging component of this project, and how they can be used. (The Monitoring component of this project, outside the scope of this article, can also be customized - see https://github.com/sassoftware/viya4-monitoring-kubernetes/blob/master/monitoring/README.md#customiz....
Note: You may already have something like this set up, from when you deployed the monitoring and logging modules. This option is perhaps a bit overcomplicated if you only wish to set one environment variable for the project. But is a good choice when you need to customize many of them.
Examples
Here is an example for a SAS Viya deployment in a viya_namespace called gelcorp. The SAS Viya Monitoring for Kubernetes logging module has already been installed in the cluster, in a logging_namespace called ops4viyalog. (ops4viya was a pre-release name for what is now the SAS Viya Monitoring for Kubernetes project). The whole GitHub project has been cloned into the current user's home directory as described earlier.
Set up Kibana role-based access for the gelcorp viya_namespace
cd ~/viya4-monitoring-kubernetes/logging/bin
export LOG_NS=ops4viyalog
./security_create_rbac.sh gelcorp
Expected output:
INFO Helm client version: 3.2.1
INFO Kubernetes client version: v1.18.6
INFO Kubernetes server version: v1.18.6
INFO Loading user environment file: /home/cloud-user/viya4-monitoring-kubernetes/logging/user.env
Creating access controls for namespace [gelcorp] [Sun Jan 10 10:24:14 EST 2021]
INFO Security role [search_index_gelcorp] created [201]
INFO Security rolemapping created between [search_index_gelcorp] and backend-role(s) [gelcorp_kibana_users]. [201]
INFO Security rolemapping created between [kibana_user] and backend-role(s) [gelcorp_kibana_users]. [200]
INFO You may see a message below about a process being killed; it is expected and can be ignored.
./security_create_rbac.sh: line 221: 15807 Killed kubectl -n $LOG_NS port-forward --address localhost svc/v4m-es-client-service :$ES_PORT > $tmpfile
Access controls created [Sun Jan 10 10:24:30 EST 2021]
============================================================================================================================
== Assign users the back-end role of [gelcorp_kibana_users] to grant access to log messages for [gelcorp] namespace and Kibana ==
============================================================================================================================
Set up a Kibana user named 'gelcorp_admin' for the gelcorp viya_namespace
cd ~/viya4-monitoring-kubernetes/logging/bin
export LOG_NS=ops4viyalog
./user.sh create --namespace gelcorp --username gelcorp_admin --password password-goes-here
Expected output:
INFO Helm client version: 3.2.1
INFO Kubernetes client version: v1.18.6
INFO Kubernetes server version: v1.18.6
INFO Loading user environment file: /home/cloud-user/viya4-monitoring-kubernetes/logging/user.env
INFO Attempting to create user [gelcorp_admin] and grant them access to namespace [gelcorp] [Sun Jan 10 10:31:33 EST 2021]
INFO User [gelcorp_admin] created. [201]
User [gelcorp_admin] added to internal user database [Sun Jan 10 10:31:34 EST 2021]
INFO You may see a message below about a process being killed; it is expected and can be ignored.
You can run user.sh create... more than once to create more users.
Delete a Kibana user named 'gelcorp_admin'
There is no need to specify the viya_namespace or password when deleting a user.
cd ~/viya4-monitoring-kubernetes/logging/bin
export LOG_NS=ops4viyalog
./user.sh delete --username gelcorp_admin
Expected output:
INFO Helm client version: 3.2.1
INFO Kubernetes client version: v1.18.6
INFO Kubernetes server version: v1.18.6
INFO Loading user environment file: /home/cloud-user/viya4-monitoring-kubernetes/logging/user.env
INFO Attempting to remove user [gelcorp_admin] from the internal user database [Sun Jan 10 10:38:42 EST 2021]
INFO User [gelcorp_admin] deleted. [200]
User [gelcorp_admin] removed from internal user database [Sun Jan 10 10:38:43 EST 2021]
INFO You may see a message below about a process being killed; it is expected and can be ignored.
It's recommended to delete all users you created for the viya_namespace before deleting the Kibana role-based access for that Viya namespace. If you do not, there will be one or more users left behind who have roles assigned to them that no longer exist. This may leave a Kibana user who appears to be set up properly, yet is unable to access any log data.
Delete Kibana role-based access controls for the gelcorp viya_namespace
cd ~/viya4-monitoring-kubernetes/logging/bin
export LOG_NS=ops4viyalog
./security_delete_rbac.sh gelcorp
Expected output:
INFO Helm client version: 3.2.1
INFO Kubernetes client version: v1.18.6
INFO Kubernetes server version: v1.18.6
INFO Loading user environment file: /home/cloud-user/viya4-monitoring-kubernetes/logging/user.env
Deleting access controls for namespace [gelcorp] [Sun Jan 10 11:02:02 EST 2021]
INFO Security rolemappings for [search_index_gelcorp] deleted. [200]
INFO Security role [search_index_gelcorp] deleted. [200]
INFO Security rolemapping deleted between [kibana_user] and backend-roles [gelcorp_kibana_users, gelcorp_kibana_ro_users]. [200]
INFO Security rolemapping deleted between [kibana_read_only] and backend-roles [gelcorp_kibana_users, gelcorp_kibana_ro_users]. [200]
INFO Security rolemapping deleted between [cluster_ro_perms] and backend-roles [gelcorp_kibana_users, gelcorp_kibana_ro_users]. [200]
INFO You may see a message below about a process being killed; it is expected and can be ignored.
./security_delete_rbac.sh: line 173: 25351 Killed kubectl -n $LOG_NS port-forward --address localhost svc/v4m-es-client-service :$ES_PORT > $tmpfile
Access controls deleted [Sun Jan 10 11:02:21 EST 2021]
===============================================================================================================
== You should delete any users whose only purpose was to access log messages from the [gelcorp] namespace ==
===============================================================================================================
When you delete the Kibana role-based access controls for a viya_namespace, there is no validation check to see if any users are still granted those role-based access controls.
Results: Kibana RBAC for one SAS Viya namespace
In another environment, I have two deployments of SAS Viya, in namespaces called gelcorp and gelinstitute.
The overall Kubernetes Admin user ('admin') can see log messages for both these Viya deployments, and from other K8s namespaces too:
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
I used the scripts to set up the RBAC for gelcorp, and created a user called gelcorp_admin, as per the examples above. The gelcorp_admin can only see log messages for the gelcorp namespace:
Extended results: Kibana RBAC for two SAS Viya namespaces
In the same environment, I then used the same scripts to set up Kibana roles and role mappings for my second Viya deployment, in the namespace gelinstitute:
cd ~/viya4-monitoring-kubernetes/logging/bin
export LOG_NS=ops4viyalog
./security_create_rbac.sh gelinstitute
I also created a second Kibana user called gelinstitute_admin to use the access controls for gelinstitute:
cd ~/viya4-monitoring-kubernetes/logging/bin
export LOG_NS=ops4viyalog
./user.sh create --namespace gelinstitute --username gelinstitute_admin --password password-goes-here
As expected, this user can only see log messages for the gelinstitute namespace:
Finally, I created a third new Kibana user called gelboth_admin, initially with access to gelcorp logs. You can create as many users as you wish with access to one set of Viya logs in this way. But I then ran the create script a second time to try to grant it access to gelinstitute logs.
cd ~/viya4-monitoring-kubernetes/logging/bin
export LOG_NS=ops4viyalog
./user.sh create --namespace gelcorp --username gelboth_admin --password password-goes-here
./user.sh create --namespace gelinstitute --username gelboth_admin --password password-goes-here
The second time, the user.sh script exits with an error, because the user already exists:
INFO Helm client version: 3.2.1
INFO Kubernetes client version: v1.18.6
INFO Kubernetes server version: v1.18.6
INFO Loading user environment file: /home/cloud-user/viya4-monitoring-kubernetes/logging/user.env
INFO Attempting to create user [gelboth_admin] and grant them access to namespace [gelinstitute] [Sun Jan 10 11:33:13 EST 2021]
RROR There was an issue creating the user [gelboth_admin]; user already exists. [200]
This makes perfect sense. But it is easy to finish setting up our gelboth_admin user, to give it access to the gelinstitute logs as well. Log into Kibana as the admin, and navigate to the Security page. It has an icon which looks like this:
Under Authentication Backends, click Internal Users Database. [caption id="attachment_42396" align="aligncenter" width="300"]
Kibana Security page, with Internal User Database button highlighted
In the Internal Users Database page, we see all three of our gel*_admin users (gelboth_admin, gelcorp_admin and gelinstitute_admin):
Notice that gelboth_admin currently has only one backend role, gelcorp_kibana_users. We want to also grant this user the gelinstitute_kibana_users backend role. So, click the Edit button for gelboth_admin. The under Backend Roles, click the 'Add backend role' button.
You can enter the name of the backend role to add by hand: gelinstitute_kibana_users. Or you can copy-and-paste it from the output of the security_create_rbac.sh script, e.g. from near the end of the line of output like this (you may have to scroll right to see it):
INFO Security rolemapping created between [search_index_gelinstitute] and backend-role(s) [gelinstitute_kibana_users]. [201]
Click the Submit button at the bottom of the page. Now, in the Internal Users Database page, our user gelboth_admin has both of our new Backend Roles:
As a result, gelboth_admin can now see log messages in Kibana for both Viya deployments, gelcorp and gelinstitute, but not for any other Kubernetes namespaces:
This works exactly as expected. I hope this will help you manage which users get access to log messages from each of your SAS Viya deployments on Kubernetes. See you next time!
Find more articles from SAS Global Enablement and Learning here.
