Someone validating a SAS Viya deployment usually aims to answer some combination of these questions:   Are basic elements of my SAS Viya deployment working? Are the components of my SAS Viya deployment which will be used the most working acceptably well? Do some specific non-functional performance tests of this SAS Viya deployment meet expectations? Are access controls in SAS Viya granting and preventing access to data, content and application functionality in the intended way? There are of course other possible aims for validation beyond these, such as test of integration with data sources or other applications, usability and accessibility tests, tests of internationalization and more. There is no one 'best' way to validate a SAS Viya deployment; a validation procedure should be tailored to a specific deployment, and to answer specific questions relevant to the business' needs. Choosing to include fewer validation tests makes the procedure faster, cheaper and potentially easier to automate. Having more validation steps to a procedure may mean it takes longer and costs more to identify, design, develop and run the tests, but when run they give a more complete view of a deployment's status.   Given all that, this post gives examples of validation steps that we use in the GEL team. It also includes some example tests which I personally don't have any need to run, but which we are sometimes asked about. The examples are not exhaustive; if you have experience of validating SAS Viya deployments and you check for something not mentioned here which you think others should check too, I would love to hear from you.     Edit 02 Oct 2023: In the context of life sciences and pharmaceuticals, Computer System Validation (CSV) is a rigorous testing and documentation process required to satisfy regulators that "the computer systems and their software operate as intended, meet user requirements, and comply with regulatory standards" (to quote Computer System Validation (CSV) in Life Sciences Part 1: Introduction to CSV - Verista). This article does not try to describe such rigorous validation in a regulated industry, and is limited to a more generally applicable set of confirmatory tests which may be useful to any SAS Viya administrator.     SAS Viya Validation Resources   SAS-provided tools documented in SAS Help Center   The SAS Readiness Service checks the status of the SAS Viya platform every 30 seconds, to determine whether it is ready for use. It checks whether SAS endpoints in Kubernetes are responding (e.g. with HTTP 200: OK responses to a REST call), and also whether the SAS Infrastructure Data Server database and OAuth authentication providers are responding. Since it is part of SAS Viya 'out of the box', it offers a universally-available way to check whether SAS Viya is fundamentally operational.   The SAS® Viya® Platform: Deployment Validation section in the SAS Viya Platform Administration documentation in SAS Help Center is a useful resource. It describes how to run the SAS Viya Platform Deployment Report, which is one tool in the SAS Viya 4 Administration Resource Kit, a toolset available in GitHub and which is practically indispensable for many deployment consultants, architects and similar who work with SAS Viya. We use tools in the viya4-ark in most of our GEL workshop collections. Running the script which generates the SAS Viya Platform Deployment Report is both easy and quick.   The SAS Viya Command-line Interface (CLI) is an essential tool for validation, as it is for many other administration activities.     SAS Viya Administration Checklist   The SAS Viya Administration Checklist has a task called Validate your SAS Viya Deployment. It proposes your validation first check whether Kubernetes is fundamentally working and has the expected nodes and namespaces. Then, it suggests some tests to validate SAS Viya, such as application URLs returning a page, SAS Studio code submission working for a trivially simple SAS code statement, presence of loaded data in CAS, some authorization tests and some validation tests of the SAS Programming Run-time which can be done using the SAS Viya CLI.   It also suggests using...   The ValidateViya pyviyatool   ValidateViya is a pyviyatool that runs a series of tests on a Viya environment, validating that it is running as expected. validateviya has a modular design, allowing for the creation of custom tests, the alteration of existing tests, and the removal of unneeded tests. See the vaildateviya User Guide in the pyviyatools project in GitHub.     Basic validation   *Taps microphone* "Is this thing on?"   The aim of basic validation is to run a quick set of tests which will reveal fundamental issues with your deployment fast. We check for things like Kubernetes not working, SAS Viya not being deployed successfully, pods not running, or users being unable to sign in. If major issues prevent a SAS Viya deployment from working at all, you will want to know about it early.   Validate the right one   If you have more than one SAS Viya deployment, a possible pre-requisite is to determine which one to validate. Which hosts, which Kubernetes cluster, which is the URL for the web applications in the deployment to be validated? It is of no use to test the health of the wrong environment, and allow that mistake to give you a false sense of security. Confirm the hostname or IP address for your Kubernetes ingress host and any other hosts you will interact with directly. If you have front-end load balancers, reverse proxies or similar, bypassing these during basic validation will simplify your testing, especially if there is any ambiguity about which deployment will really respond to requests sent to an alias.   Examples of steps in a basic 'is it on?' validation procedure   To give some examples, here are descriptions of some tests in a basic validation procedure for an environment I regularly rebuild - these tests are likely to be valid for nearly any SAS Viya deployment:   Can we make an SSH connection to a node in the Kubernetes cluster (the node we use to run kubectl and other administration tools)? Is kubectl using the correct kube config file for the cluster on which the SAS Viya deployment is running? Do we get the expected number of nodes, and are they all in the expected Ready state, when we run kubectl get nodes? Are the Kubernetes namespaces for SAS Viya, and for supporting applications such as our observability toolset present, in the results of kubectl get ns? How many pods are running in the Viya namespace? Is the number of running within an expected range (e.g. 170-190 pods for the SAS Viya software order I deployed)? How many pods are pending in the Viya namespace? Is the number of pending pods larger than some acceptable minimum? (E.g. from past experience with this environment, I know that there are sometimes 2-3 pending pods out of 180 or so, and this does not necessarily mean there is a problem. But in this environment, more than this number of pending pods usually does indicate a problem.) Is the sas-readiness pod itself reporting as being ready, indicating that its three types of check are all passing? See the SAS Readiness Service for more on this. Can you sign in to SAS Environment Manager as an administrator account? Does the SAS Viya license for this deployment expire on a date sufficiently far into the future? Can you authenticate successfully against the SAS Viya Command-line interface?     Validate frequently-used functional components   If basic validation passes, we generally want to look just a little bit more closely at some applications, to check if the components of SAS Viya that end users will use the most are working. This next stage of validation still needs to be fast, and reveal commonly-occurring issues which might not be revealed by the basic validation tests above.   Examples of steps in a functional validation procedure   Your basic functional validation tests, in your environment, may differ significantly from the following list depending on what your users do most in SAS Viya. The following descriptions are the gist of what is tested, not how I test those things. They would typically be a mix of manual and scriptable tests. For the environment I look after, basic functional validation includes steps like these:   In SAS Environment Manager's Data page, or using the sas-viya CLI's cas plugin, are any CAS tables are loaded into memory, and are they the tables we expect to be loaded into memory in CAS for this environment? Open SAS Studio. Does a compute session under the SAS Studio compute context start successfully? Can we run a trivially simple SAS statement such as proc options; run; , and does it produce normal log output without errors? Can we submit the same program successfully in SAS Batch, using the sas-viya CLI's batch plugin? Does the SAS batch job execute, and return sensible-looking SAS logs with no errors? From SAS Drive, can we open the User Activity report? This is efficient in the sense that it tests several things at once: SAS Drive is working, and we can navigate SAS Viya folders for their contents SAS Visual Analytics is working and able to render charts SAS Cloud Analytic Services (CAS) is working and able to provide data to SAS Visual Analytics The Audit service is running. By default it only loads data into the AUDIT table in CAS once per hour, so we may have to wait for it to run before we see data in the User Activity report. Can a simple predictive model be built, trained, published and validated using tools such as SAS Visual Analytics, SAS Model Studio and SAS Model Manager? A scripted process which passes through key functional steps in these applications should be completed successfully. Can we open and sign in to our chosen log monitoring application (such as OpenSearch Dashboards), and see log messages and their contextual data flowing in from the SAS Viya deployment? Are there an unusual volume of ERROR-level log messages being produced? Can we open and sign in to your metric monitoring application (such as Grafana), and see metric data flowing in from the SAS Viya deployment and its host Kubernetes cluster? Do a few example charts of metrics appear to display normal patterns of metric data, compared to what we would usually expect to see in those charts? If there are any other supporting applications deployed in your Kubernetes cluster alongside your SAS Viya deployment, are they healthy and working?     Validate non-functional performance   Unless something unusual has gone wrong, the same specification of physical and virtual compute and storage resources, with the same versions of Kubernetes and its supporting components running the same release of SAS Viya which has been configured the same way will typically perform about the same every time you deploy it. Non-functional performance validation aims to demonstrate that an environment performs adequately in a series of workload or stability tests, compared either to some benchmark result from a similar environment in the past, or compared to a business objective.   This kind of validation likely to be performed less frequently; it may only need to be done once, or a handful of times, per environment. Therefore, administrators may have less need for this type of validation to be fast or automated. Non-functional testing for processing performance, responsiveness, resilience etc. certainly takes longer to design, and also typically takes longer to perform than basic validation.   This is the aspect of validation I have least need for with the classroom environments I support, so perhaps you would like to reply to this post in the comments below to describe what you need to cover with this type of validation. From my experience of consulting roles in SAS, the following test ideas could be a good starting point:   Can I load a specific-size data set into CAS in an acceptable amount of time? How long does a specific typical program take to run? How long does a data processing job flow take to run? Does it finish successfully within an acceptable time? Can the same data load test be performed at any of the usual times it might need to run, including when the system is under user load, and still complete within an acceptable time? The 'acceptable time' here may be defined by a contract or service level agreement, or may be in comparison with some benchmark result from a previous test. How long does it take a user to sign in and open each of the applications they use most, be that a report, a programming session, a modelling tool etc.? How long does a specific typical report take to open? How long does a specific model flow take to complete? How does responsiveness vary when the system is under high real or simulated end-user workload? (Can this testing be used to define what 'high end-user workload' is, for this environment, and is that enough capacity for the organization who owns it?) Does a backup job complete successfully? Does running a backup have an unacceptable impact on the rest of the system's performance? Does a restore from backup complete successfully? For components of the deployment which are highly available, does the system react in the correct way if I intentionally stop/delete/remove e.g. one pod in the replica set or equivalent. For example. does Kubernetes successfully re-start the failed pod and resume sending traffic to it when it is running again?   I expect I missed some things that people will consider important here. If your SAS environment is subject to a service level agreement (SLA), that should define metrics that can be objectively tested to validate whether the environment meets the SLA or not. So, define performance validation tests that measure those metrics, and can be automated.   Let me know in the comments below what other types of non-functional testing you think a validation procedure should include.     Validate access controls   The Security Policy and Authorization Model tasks in the SAS Viya Administration Checklist describe how access controls should be defined and documented for a SAS Viya deployment. Follow these to an extent that is proportionate in the context of how tightly or loosely your SAS Viya environment and the data it contains need to be protected from unauthorized access.   Then, again as appropriate for the degree of rigor required, you can define a series of validation tests to perform once, occasionally or regularly to prove that users who should have access to data, content and application functionality do have that access, and that users who should not, do not.   You may find some value in tools to help you automate this. pyviyatools that might help include:   explainaccess.py - a flexible tool that can explain access controls on a SAS Viya folder, report, or for another object or a service URI, for any specific user or group, or for all users and groups who have applicable access controls getruleid.py - pass in a SAS Viya object or service URI, and a user or group, and get back the ID of a SAS Viya general authorization rule that applies to that object for that user listcaslibsandeffectiveaccess.py - for a specific caslib, or for all caslibs on all CAS servers, return a list of all effective access controls in the CAS authorization system listcastablesandeffectiveaccess.py - a somewhat flexible variation of the above for CAS tables: return a list of all effective access on all CAS tables (or a specific table) in all CAS libraries (or a specific CAS library) on all servers. Can optionally include row-level permissions and permissions on source tables. listgroupsandmembers.py - return list of all groups and all their members, optionally including their email addresses (if available in the identity service's data) listrules.py - list either all general authorization rules (typically for further processing) or rules that target a specific URI and/or mention a specific user or group testfolderaccess.py - assert whether a folder is accessible to a user or not. This tool is intended to be called with a specific folder path, principal, permission and a sense (whether access with this permission is expected to be granted or not). It will respond 'TEST PASSED' or 'TEST FAILED' with the specifics of what was tested. We run a series of tests using this tool in one of our GEL workshops, to validate folder permissions for a set of folders for a set of users and groups, and show how one folder is 'incorrectly' protected (intentionally so) and causes one test in the suite to fail. As an aside, the simpler and more regular your authorization model design is, the more easily validated it will be. If validating access controls is important in your organization, time spent in carefully designing and documenting the authorization model so that it is easily understood, clearly described and simple will be much more cost-effective to the business than a haphazardly designed and carelessly implemented authorization model. It will also be far easier to validate.   Between the four general types of validation tests described above, you should be able to get a good sense of whether a SAS Viya deployment is working, useable for the purpose it was intended to be used for, performant and correctly secured. There are certainly other kinds of validation that could be performed, but I think those are the main and most common ones. But what do you think - did I miss a kind of test that everyone ought to include when they validate SAS Viya? Let me know in the comments below.     Find more articles from SAS Global Enablement and Learning here. ... View more

In this Part 2 of a two-part series, we will install and use the pyviyatools on Windows.   The pyviyatools require the SAS Viya CLI and the trusted certificates file it requires, and of course Python - preferably python3. In this guide I chose to install pyviyatools using git from a command line, though like most of the things featured in this guide, there are other ways to get it. If you followed Part 1 of this guide, Install and use the SAS Viya CLI on Windows, much of the hard work is already done.   If you have not already read and followed Part 1, I would suggest you glance at this guide to get an idea of what it covers, and then go and follow the steps in Part 1 first.   This guide documents the concluding part of one possible ‘happy path’ for setting the SAS Viya CLI and pyviyatools up on Windows from scratch. If you have some of the components already, you may find you can skip the corresponding section. But between Part 1 of this guide and the steps below, below you can find the complete process.   Contents   In Part 2 of this guide, we will cover: Dependencies Install pyviyatools and the packages it requires Python3 Upgrade pip Python3 requests module Python3 configobj module git pyviyatools Use pyviyatools from a Windows command prompt Create a .authinfo file Authenticate using loginviauthinfo.py and your .authinfo file pyviyatools Examples   No doubt this guide could be improved. If you see areas you think are overcomplicated, wrong or unnecessary, or if you have questions about why I chose a particular command or installation method, I would love to hear from you. Leave a comment below or get in touch!   Dependencies   In this Part 2, we assume all the components from Part 1 are already present, so we only need to install the things in the 'Part 2' box in the dependency diagram. We will assume that everything outside that box (the Windows client machine, chocolatey, the SAS Viya CLI and the trustedcerts.pem certificates file which the pyviyatools also requires) are already set up and validated:  Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Install pyviyatools and the packages it requires Before proceeding with these steps, ensure you have followed the steps in Part 1 of this guide, and that the SAS Viya CLI is working from a Windows Command Prompt on your Windows machine, that it is connected to your SAS Viya deployment, and that you can authenticate successfully with a SAS Viya username and password and can run SAS Viya CLI commands successfully. To set up pyviyatools and the things it requires, we return to using PowerShell.   Python3 There are many ways to install Python on a Windows machine, and any of them should be okay if you can run python --version , from both PowerShell and Windows Command Prompt and get a Python 3.x.x version number in response. For the sake of consistency, we will install it using chocolatey here. Open a new PowerShell terminal window, and see if Python3 is already installed: python --version If Python3 is installed, you should see something like this, and you can continue to the next section, to upgrade pip: Python 3.11.4 If Python3 is not installed, you should see something like this (in red): python3 : The term 'python3' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. At line:1 char:1 + python3 --version + ~~~~~~~ + CategoryInfo : ObjectNotFound: (python3:String) [], CommandNotFoundException + FullyQualifiedErrorId : CommandNotFoundException If Python3 is not installed, run this command to install Python3 using chocolatey, which takes a few minutes: choco install python3 -y After installing python, close the PowerShell terminal, and open a new one. Then, run python --version again to verify that Python3 is installed and in your path. You should see something like this: Python 3.11.4 Upgrade pip   From a PowerShell terminal, run this to upgrade the Python package manager, pip: python.exe -m pip install --upgrade pip   Python3 requests module   From a PowerShell terminal, run this to install the Python3 requests module, a rich set of tools for making HTTP requests, e.g. for calling REST APIs like those exposed by SAS Viya: python -m pip install requests   Python3 configobj module   From a PowerShell terminal, run this to install the Python3 configobj module, a config file reader and writer: python -m pip install configobj   git Install git if it is not already installed. Git can be installed many ways; any method is probably fine so long as you end up getting a response similar to that shown below when you run git version . Since we are using chocolatey to install other packages in these instructions, we may as well pick that method for this step. See whether git is already installed. In the PowerShell terminal window, run this: git version If you see something like this, git is already installed: git version 2.41.0.windows.1 If you see something like this, git is not installed: git : The term 'git' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. At line:1 char:1 + git + ~~~ + CategoryInfo : ObjectNotFound: (git:String) [], CommandNotFoundException If git is not already installed, run this to install it: choco install git.install -y After installing git, close the PowerShell terminal, and open a new one. Then, run git version again to verify that git is installed and in your path. You should see something like this: git version 2.41.0.windows.1 pyviyatools We are finally ready to deploy pyviyatools. Clone the pyviyatools project from GitHub into C:\ViyaHome, using git. pyviyatools is a public project, so you do not need to authenticate to GitHub. Run this from the PowerShell terminal: Push-Location Set-Location Env: cd C:\ViyaHome git clone https://github.com/sassoftware/pyviyatools.git Pop-Location Verify that the pyviyatools project was cloned to C:\ViyaHome\pyviyatools: dir C:\ViyaHome\pyviyatools Partial expected output (with some files omitted): PS C:\Users\student> dir C:\ViyaHome\pyviyatools Directory: C:\ViyaHome\pyviyatools Mode LastWriteTime Length Name ---- ------------- ------ ---- d----- 7/14/2023 6:24 AM templates -a---- 7/14/2023 6:24 AM 57 .gitattributes -a---- 7/14/2023 6:24 AM 153 .gitignore -a---- 7/14/2023 6:24 AM 70 application.properties -a---- 7/14/2023 6:24 AM 6521 applyauthmodel.py -a---- 7/14/2023 6:24 AM 4857 applyfolderauthorization.py -a---- 7/14/2023 6:24 AM 4308 applyviyarules.py -a---- 7/14/2023 6:24 AM 7899 archivefiles.py -a---- 7/14/2023 6:24 AM 3656 callrestapi.py ... Some files omitted ... -a---- 7/14/2023 6:24 AM 3987 toggleviyarules.py -a---- 7/14/2023 6:24 AM 5327 unittestsadm33.sh -a---- 7/14/2023 6:24 AM 8687 unittestsadm34.sh -a---- 7/14/2023 6:24 AM 4563 unittestviya4.sh -a---- 7/14/2023 6:24 AM 3150 updatedomain.py -a---- 7/14/2023 6:24 AM 4731 updatepreferences.py -a---- 7/14/2023 6:24 AM 7843 validateviya-manual.md -a---- 7/14/2023 6:24 AM 23618 validateviya.py PS C:\Users\student> Run the pyviyatools setup.py script. This tool uses the Python3 configobj module we installed earlier to set values in a pyviyatools configuration file called application.properties . In the PowerShell terminal window, run this, substituting the location where you installed the sas-viya CLI executable for C:\ViyaHome\CLI , if it is somewhere else on your Windows machine. Recall that we installed and configured the SAS Viya CLI in Part 1 of this guide. cd C:\ViyaHome\pyviyatools ./setup.py --clilocation C:\ViyaHome\CLI --cliexecutable sas-viya Expected output: Note: updating C:\ViyaHome\pyviyatools\application.properties NOTE: configuration values set to {'sascli.location': 'C:\\ViyaHome\\CLI', 'sascli.executable': 'sas-viya'} Optional: If you want to be able to run pyviyatools without specifying the path, add the pyviyatools directory to your path. In the PowerShell terminal window, run this: [Environment]::SetEnvironmentVariable("Path",[Environment]::GetEnvironmentVariable("Path", [EnvironmentVariableTarget]::Machine) + ";C:\ViyaHome\pyviyatools", [EnvironmentVariableTarget]::Machine) Close the PowerShell terminal window, and open a new one, to pick up the change to the machine’s path environment variable. Use pyviyatools from a Windows command prompt   Create a .authinfo file   Create a .authinfo file. In the following command, I have used the username and password of a SAS Viya administrator account: user geladm , password lnxsas . Substitute the username and password of the account you wish to use by default for these values in the command below. In the PowerShell terminal window, run this $authinfo_file=$HOME+'\.authinfo' [IO.File]::WriteAllLines($authinfo_file, 'default user geladm password lnxsas') type $authinfo_file Note: We use WriteAllLines here to write the line to the .authinfo file, because when I used echo , the character encoding ended up being UTF-16, which was not understood by python3’s configObj module.   Authenticate using loginviauthinfo.py and your .authinfo file I have not had success getting this working from a PowerShell prompt yet, so here we switch to a Windows command prompt. Open a new Windows Command Prompt window, to pick up any changes we made to your path environment variable. In the command prompt, run this. (If you did not add the pyviyatools path to your path environment variable, you will need to specify the full path to loginviauthinfo.py and any other pyviyatools you run, e.g. C:\ViyaHome\pyviyatools\loginviauthinfo.py 😞 set current_namespace=gelcorp set SAS_CLI_PROFILE=%current_namespace% set SSL_CERT_FILE=%USERPROFILE%\.certs\%current_namespace%_trustedcerts.pem set REQUESTS_CA_BUNDLE=%SSL_CERT_FILE% loginviauthinfo.py Expected result (showing your profile name instead of ‘ gelcorp ’, your ingress host instead of ‘ gelcorp.rext03-0175.race.sas.com ’ and your SAS Viya user – ideally an administrator – instead of ‘ geladm ’): C:\Users\student>loginviauthinfo.py Logging in with profile: gelcorp Getting Credentials for: gelcorp.rext03-0175.race.sas.com Logging in as user: geladm Enter credentials for https://gelcorp.rext03-0175.race.sas.com: Login succeeded. Token saved. C:\Users\student> Pyviyatools Examples The results of each of the following examples is from one of the GEL workshop environments. Obviously the caslibs, members of SAS Administrators group, folder ids, launcher context details and so on will vary between SAS Viya deployments. List caslibs in your default CAS server   In the command prompt, run this (prefixed with C:\ViyaHome\pyviyatools\ or the path you to your copy of the pyviyatools, if you did not add the pyviyatools directory to your path): listcaslibs.py Expected output (caslibs shown depend on the caslibs visible to your authenticated user in your SAS Viya deployment – caslibs visible to my administrative user, geladm , in my SAS Viya deployment are shown in the output below): C:\Users\student>listcaslibs.py server,caslib cas-shared-default,CASUSER(geladm) cas-shared-default,Financial Data cas-shared-default,Formats cas-shared-default,hrdl cas-shared-default,ModelPerformanceData cas-shared-default,Models cas-shared-default,Public cas-shared-default,salesdl cas-shared-default,Samples cas-shared-default,SystemData C:\Users\student> List members of the SAS Administrators group   In the command prompt, run this: listgroupsandmembers.py --id SASAdministrators Expected output: groupid,groupname,grouptype,groupproviderid,memberid,membername,membertype,memberproviderid SASAdministrators,SAS Administrators,group,local,geladm,"geladm",user,ldap SASAdministrators,SAS Administrators,group,local,GELCorpSystemAdmins,"GELCorp System Admins",group,ldap Get the folderid of the /Public folder   In the command prompt, run this: getfolderid.py --f /Public Expected output: creationTimeStamp ,createdBy ,modifiedTimeStamp ,modifiedBy ,version ,id ,name ,description ,type ,memberCount ,properties , "2023-07-13T08:08:51.478982Z","anonymous","2023-07-13T08:08:51.478982Z","anonymous","1","fef7a224-d3fd-4c03-8a51-fdd05b0cbfe0","Public","Public folder for general access.","folder","0","{'allowMove': 'false', 'uuid': '935842da-9ba0-479c-99e3-6a4e489322fc'}", Compare this output with that from the sas-viya folders show command: sas-viya --output text folders show --path /Public Explain the access controls on the /Public folder   In the command prompt, run this: explainaccess.py -f /Public --header Expected output: principal,read,update,delete,secure,add,remove,read(convey),update(convey),delete(convey),secure(convey),add(convey),remove(convey) authenticatedUsers,grant*,prohibit,prohibit,prohibit,grant*,grant*,grant*,grant*,grant*,prohibit,grant*,grant* SASAdministrators,grant,grant,grant,grant,grant,grant,grant,grant,grant,prohibit,grant,grant Validate your SAS Viya platform is working   In the command prompt, run this: validateviya.py Expected output: Data Collection Test Started: Logged in User Data Collection Test Started: List Users Data Collection Test Started: List Base Folders Data Collection Test Started: List CAS Servers Data Collection Test Started: List CAS Server Metrics Data Collection Test Started: List CAS Server Caslibs Data Collection Test Started: List CASLib Tables Computation Test Started: Run Test SAS Code name ,id "geladm","geladm" name ,id "SAS LDAP Service Account","sasldap" "SAS System Account","sas" "CAS System Account","cas" "SAS Administrator","sasadm" "SAS Test User 1","sastest1" "SAS Test User 2","sastest2" "geladm","geladm" "Douglas","Douglas" "Delilah","Delilah" "Alex","Alex" "Amanda","Amanda" "Ahmed","Ahmed" "Fay","Fay" "Fernanda","Fernanda" "Fiona","Fiona" "Frank","Frank" "Fred","Fred" "Hamish","Hamish" "Hazel","Hazel" "Heather","Heather" "Helena","Helena" "Henrik","Henrik" "Hugh","Hugh" "Santiago","Santiago" "Sarah","Sarah" "Sasha","Sasha" "Sean","Sean" "Sebastian","Sebastian" "Shannon","Shannon" "Sheldon","Sheldon" "Sophia","Sophia" "hrservice","hrservice" "salesservice","salesservice" "financeservice","financeservice" name ,description "Users","Base Folder for all user folders." "Public","Public folder for general access." "Products","Products folder for samples, canned objects and other SAS-provided content." "gelcontent","Base folder" "Conversational Flows","Conversational Flows Folder" "Model Repositories","None" name ,host ,port ,description "cas-shared-default","sas-cas-server-default-client","5570","controller" serverName ,systemNodes ,systemCores ,cpuSystemTime ,memory "cas-shared-default","4","32","0.013651","1319040" name ,scope ,description "CASUSER(geladm)","global","Personal File System Caslib" "Financial Data","global","gelcontent finance" "Formats","global","Stores user defined formats." "hrdl","global","gelcontent hrdl" "ModelPerformanceData","global","Stores performance data output for the Model Management service." "Models","global","Stores models created by Visual Analytics for use in other analytics or SAS Studio." "Public","global","Shared and writeable caslib, accessible to all users." "salesdl","global","gelcontent salesdl" "Samples","global","Stores sample data, supplied by SAS." "SystemData","global","Stores application generated data, used for general reporting." serverName ,caslibName ,name "cas-shared-default","systemData","AUDIT" "cas-shared-default","systemData","AUDIT_ACTIVITIES" "cas-shared-default","systemData","AUDIT_ACTIVITIES_PROPERTIES" "cas-shared-default","systemData","AUDIT_ACTIVITIES_URIS" "cas-shared-default","systemData","SASVIYATYPES" runSuccessful ,jobState "True","completed" From here on, you should be able to run sas-viya commands and run any of the pyviyatools as permitted by the versions of the software you have available, and the permissions of the user you defined in your .authinfo file. The SAS Viya CLI has an extensive set of plugins, documented in SAS Help Center here: https://go.documentation.sas.com/doc/en/sasadmincdc/default/calcli/n1vth8mtb8ipprn1prz5j26p3nvc.htm.   pyviyatools provides many convenient tools/scripts which my GEL colleague Gerry Nelson and I, and a number of other contributors have written to enable or simplify certain tasks in SAS Viya. It is documented in markdown files in the pyviyatools project in GitHub, particularly in: README.md: https://github.com/sassoftware/pyviyatools/blob/master/README.md INSTALL.md: https://github.com/sassoftware/pyviyatools/blob/master/INSTALL.md loginviauthinfo.md describes how to use loginviauthinfo.py, one of our most useful tools, to authenticate to SAS Viya before running either pyviyatools or SAS Viya CLI commands: https://github.com/sassoftware/pyviyatools/blob/master/loginviauthinfo.md EXAMPLES.md: https://github.com/sassoftware/pyviyatools/blob/master/EXAMPLES.md SUPPORT.md: https://github.com/sassoftware/pyviyatools/blob/master/SUPPORT.md CONTRIBUTING.md: https://github.com/sassoftware/pyviyatools/blob/master/CONTRIBUTING.md All the tools I have tried in the course of writing this guide worked well on Windows, as they are supposed to. However, due to the nature of our work, my fellow pyviyatools developers and I use them in LINUX more than in Windows. The tools are supposed to work the same on both but it is possible that you may find a tool that does not work as well on Windows as it does in LINUX. Please let us know about any issues that you come across while using the pyviyatools, by raising an issue in the pyviyatools project or by emailing one of us. It would help if you mention that you are using it in Windows, if you are. My thanks to Gerry Nelson and Allan Tham for their help in discussing and refining some of the things I included in this guide, and to Gerry in particular for his work in developing and publishing the pyviyatools.   See you next time!   Find more articles from SAS Global Enablement and Learning here. ... View more

In this Part 1 of a two-part series, we will go through a beginning-to-end process to install, configure and use the SAS Viya Command-Line interface on Windows. In Part 2, we will install and use the pyviyatools on Windows, which will be a little simpler, once we have done the hard work of completing Part 1.   Why do we need guides like this? Is it not all documented already? Well, yes... but...   In the SAS Global Enablement and Learning team, we (and people attending our workshops) normally install and use both the SAS Viya CLI and pyviyatools on Linux machines. Even if we want to run these tools from our own Windows laptop or desktop, we generally find it convenient to do so from a Bash shell running in Windows Subsystem for Linux. If this is an option for you, it is really nice. Installation of these tools in Linux is comparatively straightforward, and seems much easier to script than it is on Windows.   However, customers whose end users have Windows machines often want to install the SAS Viya CLI and pyviyatools on those Windows machines. Both tool sets do run on Windows, but beginning-to-end instructions for how to set up all the various components and dependencies are hard to find. It seems you have to piece them together from documentation in several different places, which is tedious.   This guide is therefore one possible ‘happy path’ for setting these tools up on Windows from scratch. There are a LOT of steps, which may be part of the reason why there are few other guides to the whole procedure. If you have some of the components already, you may find you can skip the corresponding section. But below you can find the complete process. I hope it turns out to be useful.   Contents   In Part 1 of this guide, we will cover: Dependencies Prerequisites Windows Client Machine PowerShell and Windows Command Prompt are both required Installation directories Install the SAS Viya CLI Installing the SAS Viya CLI is difficult to automate Download SAS Viya CLI using a web browser Install SAS Viya CLI and its plugins Enable SAS Viya CLI to communicate with SAS Viya Chocolatey curl kubectl kubeconfig file Certificates file Define SAS Viya CLI profiles Validate that sas-viya CLI can access your SAS Viya deployment Use the SAS Viya CLI from a Windows command prompt SAS Viya CLI Examples No doubt this guide could be improved. If you see areas you think are overcomplicated, wrong or unnecessary, or if you have questions about why I chose a particular command or installation method, I would love to hear from you. Leave a comment below or get in touch!   Dependencies   The following diagram shows how the packages required to use the SAS Viya CLI and pyviyatools depend on each other, for installation and at run time, and how some of them depend on prerequisites.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The green boxes are things that we will install if they are not already installed. Really, all the things in green boxes require a Windows client machine, but showing that would not make the diagram any more clear.   The SAS Viya CLI has been carefully designed not to have much in the way of dependencies. However, if you have enabled Transport Layer Security for your SAS Viya platform, the Windows machine where the SAS Viya CLI runs must trust the certificates presented by the SAS Viya server in order to communicate with it. This is achieved by providing it with a bundle of trusted CA certificates, shown as ‘certificates file’ in the diagram. SAS Help Center's Command-Line Interface: Preliminary Instructions explain this in more detail. They also explain how to obtain that file using kubectl and a kubeconfig file, steps which are a part of this guide.   Since our goal is to install both the SAS Viya CLI and pyviyatools, we may as well use kubectl (a pre-requisite of pyviyatools) to get the necessary certificate file ourselves, rather than just having you provide it as a prerequisite.   Your Windows machine may already have some of these tools installed. If you find you already have a component like python3 or git, or you prefer to install it some other way, that is fine. You may need to adapt some of the installation instructions below to adapt for differences between your machine and the more-or-less new Windows VM I used to write the instructions.   Prerequisites   Pre-requisites are shown as grey boxes in the dependency diagram. This article does not cover how to create: A SAS Viya deployment, in a Kubernetes cluster. Your Kubernetes administrator, SAS Administrator, deployment consultant or service provider sets this up. A kubeconfig file is required to deploy SAS Viya. You could use this kubeconfig file, or another one that your Kubernetes administrator provides, whichever they prefer. When SAS Viya is deployed, a trustedcerts.pem certificates file is created automatically as part of that process, and is stored by the SAS Logon service. No additional steps are required to create this, and we will get a copy of it during the process described in this guide. A username and password that allow you to authenticate to your SAS Viya deployment, preferably as a SAS Administrator. Get this from your SAS Administrator. A Windows client machine on which you can install the chocolatey package manager, curl, kubectl and the sas-viya CLI, and if you continue to part2 (coming soon), python3 and its requests and configobj modules, git and finally pyviyatools. Windows Client Machine   You will need a Windows machine to try installing SAS Viya CLI and pyviyatools. Use your own Windows desktop/laptop if you like and have sufficient permissions.     PowerShell and Windows Command Prompt are both required   I have chosen to use both PowerShell and Windows Command Prompt for this guide, since each is required for certain tasks.   PowerShell has better commands (cmdlets) for things like setting environment variables and unzipping .zip archives, and it is likely the shell a modern Windows administrator will be more familiar with. However, I have not yet discovered how to (or if you can) run the sas-viya CLI in PowerShell.   I found fewer examples showing how to install packages we need using Windows Command Prompt, so it would take more work to convert these instructions to run solely in a Command Prompt, though I think you could. The SAS Viya CLI and pyviyatools both work in a Command Prompt, and I think it is arguably more familiar to non-Windows administrators.   I would love to hear from you if: you know how to get the SAS Viya CLI working in PowerShell (or if you know for sure that you can’t) you can convert all steps in this guide to work in Windows Command Prompt without too much additional complexity   Installation directories   These instructions put kubectl, a kubeconfig file, the sas-viya CLI executable, and pyviyatools in the following directories: Tool or file Directory Kubectl C:\Program Files\kubectl Kubeconfig file $HOME\.kube\config SAS Viya certificates $HOME\.certs SAS Viya CLI executable C:\ViyaHome\CLI pyviyatools C:\ViyaHome\pyviyatools But none of them have to be in those directories. You are free to put any of these things in a different directory, as required by your Windows machine’s disk space or permission constraints, or to suit your preferences. Modify references to the path for each component in e.g. environment variables accordingly.   Install the SAS Viya CLI   In this Part 1 of the guide, we will install the SAS Viya CLI, but also install chocolatey, curl and kubectl, and provide a kubeconfig file so that we can get the certificates file that the SAS Viya CLI needs to trust TLS certificates presented by SAS Viya. If your goal is to set up the SAS Viya CLI only, and not pyviyatools, you can stop at the end of part 1.   Other readers may want to continue on to part 2, to install pyviyatools on Windows. Since pyviyatools depends on the SAS Viya CLI, and because in part 2. I chose to use the chocolatey package manager to install python and git, you should complete part 1 of this guide first even if your main aim is to install pyviyatools.   Preliminary set-up instructions for the SAS Viya CLI can be found in the SAS Help Center documentation at: https://go.documentation.sas.com/doc/en/sasadmincdc/default/calcli/n1e2dehluji7jon1gk69yggc6i28.htm   Installing the SAS Viya CLI is difficult to automate   Unless you have a tool that can navigate to a web page in a browser and click a button in that web page automatically, you cannot fully automate installation of the SAS Viya CLI. Such tools are commonly used to functionally test web pages (among other things), but most of us don’t have one, and it doesn’t make sense to spend the money or effort setting one up just to script this one step.   Without such a tool, you must either: click a button on a web page popup dialog in your browser, in order to accept the ‘SAS License Agreement for Download’, or download the SAS Viya CLI executable ahead of time, and provide it e.g. in a volume accessible to scripts which you can run on your Windows client machine. We prefer option '2' above for our workshop environments. For these instructions, option '1' above is simpler.     Download SAS Viya CLI using a web browser   In a web browser in your Windows client machine, search the web for "SAS Viya CLI download" or browse to https://support.sas.com/downloads/package.htm?pid=2512. I used Chrome on my machine, but any browser should work fine. Sign in to sas.com. In the web page that opens once you have signed in, find the download link for ‘sas-viya for Windows’:   At the time of writing this was sas-viya-cli-1.21.10-windows.zip, but this is updated quite regularly; you should choose the current version. Click the link to download the .zip file.   Read the ‘SAS License Agreement for Download’ and click the ‘Accept’ button if you accept it:   Note: This step cannot currently be automated without software which I think is too expensive or complex to recommend for this use case alone.   Since I used Chrome as my browser, the .zip file was automatically downloaded to $HOME\Downloads (aliased as %USERPROFILE%\Downloads if you are using Windows Command Prompt). Other browsers may place the .zip file in another location by default. If they do not save it to your Downloads directory by default, Save As, move or copy it there yourself.   Install SAS Viya CLI and its plugins   Open a new PowerShell terminal window. It may be convenient to pin it to the Windows Task Bar or Start Menu. In your PowerShell terminal window, and run this to create a directory to contain the extracted SAS Viya CLI executable, unzip the downloaded file to $HOME\Downloads\sas-viya-cli, and to then copy the sas-viya.exe executable to C:\ViyaHome\CLI: mkdir C:\ViyaHome\CLI $files = Get-ChildItem $HOME\Downloads\sas-viya-cli*.zip | select-object -first 1 foreach($f in $files){Expand-Archive $f -DestinationPath C:\ViyaHome\CLI} Get-ChildItem C:\ViyaHome\CLI Tip: Scroll the textbox above to the right if necessary.   Run this to install the sas-viya CLI plugins – they are automatically installed to a new directory, at $HOME\.sas: cd C:\ViyaHome\CLI .\sas-viya plugins install --repo SAS all Add the sas-viya CLI directory path to your machine’s Path environment variable. In the PowerShell terminal window, run this: [Environment]::SetEnvironmentVariable("Path",[Environment]::GetEnvironmentVariable("Path", [EnvironmentVariableTarget]::Machine) + ";C:\ViyaHome\CLI", [EnvironmentVariableTarget]::Machine) Tip: Scroll the textbox above to the right if necessary, to see the whole command.   Close the PowerShell terminal window. We will open a new one in the next step, to pick up the change to the machine’s path environment variable.   Enable SAS Viya CLI to communicate with SAS Viya   We will need to provide a certificates file and set some environment variables before we can use the SAS Viya CLI to connect to our SAS Viya deployment. To get the certificates file, we will set up kubectl and provide a kubeconfig file (see Prerequisites  above) for the Kubernetes cluster where the SAS Viya platform is running. This set of steps is performed in PowerShell.   Chocolatey   Chocolatey is an open-source command-line package manager for Windows. We will use it to install packages in both Part 1 and part 2 of this guide.   The following instructions are based on Chocolatey Software Docs | Setup / Install (https://docs.chocolatey.org/en-us/choco/setup). With the Windows VM I used to develop these instructions, the first attempt at installation results in an error message asking you to restart the machine, and the second attempt at installation succeeds. Open a new Windows PowerShell terminal window. Run this command: Get-ExecutionPolicy If the response is Unrestricted , continue to next step. If the response is Restricted , then run one of these commands: Set-ExecutionPolicy AllSigned or Set-ExecutionPolicy Bypass -Scope Process Run this to install Chocolatey. Installation takes several minutes, and at one stage there will be no new console output for several (5-10) minutes: Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1')) Tip: Scroll the textbox above to the right if necessary, to see the whole command. If the previous command finishes by showing the message below (in red – but don’t panic, it’s okay!), do what the message says and reboot your Windows machine: .NET Framework 4.8 was installed, but a reboot is required. Please reboot the system and try to install/upgrade Chocolatey again. At C:\Users\student\AppData\Local\Temp\2\chocolatey\chocoInstall\tools\chocolateysetup.psm1:815 char:11 + throw ".NET Framework 4.8 was installed, but a reboot is re ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : OperationStopped: (.NET Framework ...ocolatey again.:String) [], RuntimeException + FullyQualifiedErrorId : .NET Framework 4.8 was installed, but a reboot is required. Please reboot the system and try to install/upgrade Chocolatey again. Once your Windows machine has restarted, sign in again, and open a PowerShell terminal window again. Re-run the command above to try installing Chocolatey again. This time it should complete in a few seconds, and be successful:   curl   Run this command in the same PowerShell window to install curl: choco install curl -y kubectl See Install and Set Up kubectl on Windows | Kubernetes for the latest instructions. Create a directory for the kubectl executable. It does not have to be in C:\Program Files\kubectl, but that path seems logical to me. In the PowerShell terminal window, run this: mkdir "C:\Program Files\kubectl" Install kubectl. In the PowerShell terminal window, run this. You should cd to the directory you created in the previous step. And this curl command fetches the current version of kubectl (1.27.2) at the time I wrote this guide. See Install and Set Up kubectl on Windows | Kubernetes for the latest version, as new versions are released frequently. You may need to find the version of Kubernetes your cluster is running, and use a compatible version of kubectl. cd "C:\Program Files\kubectl" curl.exe -LO https://dl.k8s.io/release/v1.27.2/bin/windows/amd64/kubectl.exe Validate kubectl. In the PowerShell terminal window, run this: & "C:\Program Files\kubectl\kubectl.exe" version --client --output=yaml Expected output, showing the version of kubectl you just installed: PS C:\Program Files\kubectl> & "C:\Program Files\kubectl\kubectl.exe" version --client --output=yaml clientVersion: buildDate: "2023-05-17T14:20:07Z" compiler: gc gitCommit: 7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647 gitTreeState: clean gitVersion: v1.27.2 goVersion: go1.20.4 major: "1" minor: "27" platform: windows/amd64 kustomizeVersion: v5.0.1 PS C:\Program Files\kubectl> Add the kubectl directory path to your machine’s Path environment variable. In the PowerShell terminal window, run this, if necessary substituting the full path to the directory where you installed kubectl for C:\Program Files\kubectl if you installed it somewhere else: [Environment]::SetEnvironmentVariable("Path",[Environment]::GetEnvironmentVariable("Path", [EnvironmentVariableTarget]::Machine) + ";C:\Program Files\kubectl", [EnvironmentVariableTarget]::Machine) Tip: Scroll the textbox above to the right if necessary, to see the whole command. Close the PowerShell terminal window and open a new one, to pick up the change to the machine’s path environment variable. Test that kubectl is now in your path. Run this from any directory in the PowerShell command window (it should also work from a Windows command prompt): kubectl version --client --output=yaml Expected output, showing the version of kubectl you just installed: PS C:\Users\student> kubectl version --client --output=yaml clientVersion: buildDate: "2023-05-17T14:20:07Z" compiler: gc gitCommit: 7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647 gitTreeState: clean gitVersion: v1.27.2 goVersion: go1.20.4 major: "1" minor: "27" platform: windows/amd64 kustomizeVersion: v5.0.1 PS C:\Users\student> kubeconfig file   Create a directory for the kubeconfig file. In the PowerShell terminal window, run this: mkdir $HOME\.kube Copy your kubeconfig file (see Prerequisites above) to the file path and filename $HOME\.kube\config . Make sure the file does not have a filename extension such as .txt . The kubeconfig file should be named simply config . Tips:   Get a kubeconfig file from the Kubernetes cluster administrator who maintains the Kubernetes cluster that SAS Viya is deployed into. If you want to give it another name, or put it in another directory, you can define an environment variable called KUBECONFIG and give it the value of the path to the kubeconfig file you want to use, and then kubectl will use that config file. This is useful if you have more than one kubeconfig file and wish to switch between them. You can specify a kubeconfig file on the kubectl command line ( kubectl --kubeconfig path ), but if becomes a bit cumbersome and clutters up the command, so I prefer not to do that. Run this from the PowerShell terminal prompt to view the contents of the kubeconfig ( config ) file to verify it is in the expected location: type $HOME\.kube\config Here is a screenshot showing the content of a config file (with the certificate data obscured). Yours should resemble this, though the server, cluster, user and context names will differ:       Validate that you now have access to your Kubernetes cluster using kubectl and your kubeconfig file. Run this from any directory in the PowerShell command window (it should also work from a Windows command prompt): kubectl get ns Expected output is a list of the namespaces in your Kubernetes cluster, e.g. here in PowerShell, but the output is almost identical from a command prompt. We will switch to using a command prompt later, when we use the sas-viya CLI: PS C:\Users\student> kubectl get ns NAME STATUS AGE argo Active 10d cert-manager Active 10d default Active 10d gelcorp Active 10d kube-node-lease Active 10d kube-public Active 10d kube-system Active 10d nfs Active 10d nginx Active 10d v4mlog Active 10d v4mmon Active 10d PS C:\Users\student> One of the Kubernetes namespaces should be the namespace where SAS Viya is deployed – in the example above, that namespace is gelcorp .   Note: It is useful to be able to connect to more than one SAS Viya deployment. So, I prefer to include the namespace name in any configuration variables or file names which refer to that SAS Viya deployment. Therefore, from here on in this guide, wherever you see the string gelcorp in a variable or file name, you should substitute the name of your SAS Viya namespace into that variable or filename, instead of gelcorp .   Certificates file   In this step, we copy a trustedcerts.pem certificates file from the SAS Logon service in your SAS Viya deployment to allow the SAS Viya CLI to trust the TLS certificates that SAS Viya will present to it. If you have not enabled TLS for communication between your SAS Viya deployment and external clients (browsers and the CLI), and you access SAS Viya using http, not https, you may be able to skip this step. Create a directory for the certificates. In the PowerShell terminal window, run this: mkdir $HOME\.certs Optional: See the name of your SAS Viya deployment's sas-logon-app pod. Remember to replace gelcorp in the command which follows with the name of your SAS Viya deployment’s namespace. In the PowerShell terminal window, run this: kubectl -n gelcorp get pod -l 'app=sas-logon-app' Store the sas-logon-app pod name in a variable: $SAS_LOGON_APP_POD_NAME=$(kubectl -n gelcorp get pod -l 'app=sas-logon-app' | select-string sas-logon-app | Foreach-Object { $_.ToString() -split '\s+'} | select-object -first 1) echo $SAS_LOGON_APP_POD_NAME Copy the trustedcerts.pem file from the sas-logon-app container in the sas-logon-app pod to a file named gelcorp_trustedcerts.pem in the .certs directory you just created. Run this in PowerShell, substituting all occurrences of gelcorp for your namespace name: Push-Location Set-Location Env: cd $HOME kubectl -n gelcorp cp $SAS_LOGON_APP_POD_NAME":security/trustedcerts.pem" .certs\gelcorp_trustedcerts.pem -c sas-logon-app Pop-Location Note: The code above can surely be improved. I had trouble concatenating the destination path $HOME\.certs\gelcorp_trustedcerts.pem in the command, so I had to change directory instead, and for neatness change back afterwards. Let me know if you see a neater way to do that! Verify you got the certs file, again substituting gelcorp for your namespace name: type $HOME\.certs\gelcorp_trustedcerts.pem Expected results – LOTS of output!     Define SAS Viya CLI profiles   Optional: see details of the ingress defined for your Viya deployment’s sas-logon-app service. Remember to replace gelcorp in the command which follows with the name of your SAS Viya deployment’s namespace. In the PowerShell terminal window, run this: kubectl -n gelcorp get ing | findstr sas-logon-app Expected output is something resembling this, showing in this example that the ingress hostname is gelcorp.pdcesx03038.race.sas.com : PS C:\Users\student> kubectl -n gelcorp get ing | findstr sas-logon-app sas-logon-app nginx gelcorp.pdcesx03038.race.sas.com,*.gelcorp.pdcesx03038.race.sas.com 10.43.17.23 80, 443 10d Get the ingress URL and store it in a shell variable. In the PowerShell terminal window, run this – you might need to modify it to fit the string returned from kubectl get ing on your environment, and again, remember to replace both occurrences of gelcorp in the command which follows with the name of your SAS Viya deployment’s namespace: $INGRESS_URL=$(kubectl -n gelcorp get ing | select-string sas-logon-app | Foreach-Object { $_.ToString() -split '\s+'} | select-string gelcorp |%{$_ -split ','} | select-object -first 1) echo $INGRESS_URL In your PowerShell terminal window, create a default sas-viya CLI profile which will be used if you don't specify which profile to use when calling the sas-viya executable, and one for your SAS Viya namespace, e.g. mine is called gelcorp , but you should name yours for your SAS Viya namespace. Specify http or https as appropriate for the endpoint: sas-viya.exe --profile Default profile set-endpoint https://$INGRESS_URL sas-viya.exe --profile Default profile toggle-color off sas-viya.exe --profile Default profile set-output fulljson sas-viya.exe --profile gelcorp profile set-endpoint https://$INGRESS_URL sas-viya.exe --profile gelcorp profile toggle-color off sas-viya.exe --profile gelcorp profile set-output fulljson Validate that sas-viya CLI can access your SAS Viya deployment   Now you have everything you need to access a SAS Viya deployment using the SAS Viya CLI. Let’s validate that we can, using the Windows Command Prompt.   Note: This step uses Windows Command Prompt, not PowerShell! Open a Windows Command Prompt terminal window. It may be convenient to pin it to the Windows Task Bar or Start Menu. In your Windows Command Prompt set environment variables for using the sas-viya CLI. As before, substitute the name of your SAS Viya deployment's namespace for gelcorp : set current_namespace=gelcorp set SAS_CLI_PROFILE=%current_namespace% set SSL_CERT_FILE=%USERPROFILE%\.certs\%current_namespace%_trustedcerts.pem set REQUESTS_CA_BUNDLE=%SSL_CERT_FILE% Run this (from any directory) in your Windows Command Prompt to log in to your SAS Viya deployment from the SAS Viya CLI. Provide the username and password of the account you will use to login to SAS Viya (see Prerequisites above), in place of username geladm password lnxsas in the example below: sas-viya auth login -u geladm -p lnxsas Expected output is something like this: C:\Users\student>sas-viya auth login -u geladm -p lnxsas Enter credentials for https://gelcorp.pdcesx03038.race.sas.com: Login succeeded. Token saved. C:\Users\student> You should now be able to run sas-viya commands to do anything that username is authorized to do. Try using the SAS Viya CLI to get information about the SAS Administrators group: sas-viya identities list-groups --id SASAdministrators Expected output is something like this (with some lines omitted): C:\Users\student>sas-viya identities list-groups --id SASAdministrators { "items": [ { "creationTimeStamp": "2023-07-13T08:15:30.881Z", "description": "Provides assumable access throughout the general authorization system.", "id": "SASAdministrators", "links": [ --- some lines omitted--- ], "modifiedTimeStamp": "2023-07-13T08:15:30.881Z", "name": "SAS Administrators", "providerId": "local", "state": "active", "type": "group", "version": 1 } ] } If you see JSON output similar to that above, your SAS Viya CLI has successfully authenticated and run a command against your SAS Viya deployment. Let’s try a few more examples.   Use the SAS Viya CLI from a Windows command prompt   Note: This step uses Windows Command Prompt, not PowerShell!   SAS Viya CLI Examples   There are lots of other videos and blog posts describing use of the sas-viya CLI. A search here, or using your favorite web search engine will produce many results, and I'd encourage you to explore topics, tasks and plugins that seem interesting to you. It is a fantastic set of tools! But here, I'll pick some examples that seem interesting to me - and I hope to you.   The results of each of the following examples is from one of the GEL workshop environments. Obviously the values returned will vary between SAS Viya deployments, so expect your results to differ from mine, and go explore what you can do!   See the usage help text for the sas-viya compute contexts command All sas-viya plug-ins, including the compute plug-in are designed to be self-documenting. Add a -h at the end of any partial command to see what sub-commands, command options and arguments it can take. This is a great way to discover what they can do. In the command prompt, run this to see usage for the sas-viya compute contexts command: sas-viya compute contexts -h Expected output (other versions of the plugin may offer different options and arguments to these): NAME: sas-compute-cli contexts - Manages the compute contexts. USAGE: sas-compute-cli contexts command [command options] [arguments...] VERSION: 1.37.8 COMMANDS: create Creates a compute context. delete, del Deletes a compute context by name, ID, or ordinal. generate-template, gen Generates a template to create a compute context. help, h Shows a list of commands or help for one command. list Lists the compute contexts' names and IDs. show Shows a compute context by name, ID, or ordinal. show-template, show-t Shows a template that is used to create a compute context. validate, val Validates the compute context session. OPTIONS: --help, -h Shows help. List compute contexts In the command prompt, run this to list compute contexts currently defined in your SAS Viya deployment (and that your current user is authorized to see): sas-viya compute contexts list Expected output (your compute contexts may differ to these, and the IDs will be different): 1. SAS Studio compute context c4eeed2e-2c5e-48ee-ac69-9c82889a4081 2. SAS Job Execution compute context 14f5cc90-bd7c-4306-bee2-830cd6c34bee 3. CAS Formats service compute context 4c786325-25d0-46e6-81d7-038d3bc9efa7 4. CAS Management service compute context 989ba5c6-2ede-4011-90e2-d69110aa2d6d 5. SAS Model Manager compute context d731a2dd-249b-4301-9296-31ef8364049e 6. Import 9 service compute context 695b9e48-c0cf-4be5-bd4b-be1e8d3c5cb4 7. Natural Language Conversations service compute context f23a7bc4-78b4-48a8-bfab-f6ee184edc97 8. Data Mining compute context f3ec8041-bf8b-46a0-9805-2ab9c1f50a76 9. SAS Backup job compute context e436717f-cd5c-43fe-9231-c2d805692b62 List the details of the SAS Studio Compute Context In the command prompt, run this to view details of the SAS Studio Compute Context. You can indicate which compute context you want to see by ordinal number, name or id - here I chose to identify it by name: sas-viya compute contexts show --name "SAS Studio compute context" Expected output (some lines omitted for brevity): { "createdBy": "sas.studio", "creationTimeStamp": "2023-07-13T08:18:09.70045Z", "description": "Compute context to be used by the SAS Studio service", "environment": {}, "id": "c4eeed2e-2c5e-48ee-ac69-9c82889a4081", "launchContext": { "contextName": "SAS Studio launcher context" }, "launchType": "service", "links": [ --- some lines omitted --- ], "modifiedBy": "sas.studio", "modifiedTimeStamp": "2023-07-13T08:18:09.70045Z", "name": "SAS Studio compute context", "version": 4 } If your objective was only to set up the SAS Viya CLI, and not pyviyatools, you are finished, and I hope this process has been useful.   No doubt this guide could be improved. If you see areas you think are overcomplicated, wrong or unnecessary, or if you have questions about why I chose a particular command or installation method, I would love to hear from you. Please leave a comment below or get in touch.   If you want to continue to set up pyviyatools, look no further than part 2 Install and use the pyviyatools on Windows.    My thanks to my colleagues Gerry Nelson and Allan Tham for their help in discussing and refining some of the things I included in this guide.   See you next time!   Find more articles from SAS Global Enablement and Learning here. ... View more

In this post I would like to highlight a neat way to set a useful bit of SAS Viya Programming Run-time Environment configuration from the command-line, and in the process remind you that pre-started (and reusable) compute servers are a good thing for SAS administrators to know about.   SAS Viya's Programming Run-time Environment has the ability to pre-start a pool of available compute servers. The idea is that when users request a compute session in a specific compute context in e.g. SAS Model Studio or SAS Studio, the compute session starts in a few seconds rather than a few tens of seconds because the compute session runs in a compute server that is already started. The user doesn't need to wait for the SAS Compute Service to ask SAS Launcher Service to launch a new Compute Server pod, wait for the pod to start up, and wait for all the containers in it to initialize. Those things still have to happen, but they happen before the user requests their compute session.   The apparent improvement in session start time is especially noticeable when you run something like a model pipeline which starts several compute sessions in quick succession.   The necessary steps to create a compute context which pre-starts compute servers are documented in: Configure Compute Servers to Run under a Shared Account Configure Reusable Compute Servers Configure a Pool of Available Compute Servers However, if you like to script things so that e.g. your environment creation can be automated, read on. We're going to look at a way to enable a pool of pre-started compute servers with just three commands. Plus, the method we'll use can easily be modified to create other compute contexts with attributes and restricted access.   We have covered pre-started compute servers before. 18 months ago my GEL colleague Rob Collum wrote here about improving concurrency performance in SAS Viya. One of the things he covered was how to create a pool of reusable compute servers, and he showed how to do it using the SAS Environment Manager's web interface. And 14 months ago I used the settings we'll review today as examples in my post Add, update and remove compute context attributes with new pyviyatools. But it is a while since we've written about pre-started compute servers here, so perhaps they are worth mentioning again.   First, let's refresh our memories with a few reminders about pre-started compute contexts: A compute context you configure to have pre-started available compute servers must run those compute servers under a shared service account You must store credentials for that shared service account in advance using the compute service's plugin to the sas-viya CLI If you are running a version of SAS Viya earlier than version 2023.03 (Stable or LTS), then as Rob mentioned in his post, you cannot do this if your users identities are provided to SAS Viya by SCIM, you can only do this if they are provided by LDAP. However, once you have upgraded to SAS Viya version 2023.03 (stable or LTS) or later, deployments using SCIM can now also use shared service accounts to run sessions for SAS Compute Server (and other things). Only compute servers can be pre-started, you cannot pre-start batch or connect servers Each pre-started compute server pod consumes a bit of CPU and memory on the host nodes in your Kubernetes compute node pool, even when it is idle. Not only that, even when those pods are idle and not actually using much CPU and memory, Kubernetes effectively reserves an amount of CPU and memory for their use as specified by their container's CPU and memory 'requests' values. Having pre-started compute servers may result in Kubernetes being unable to run as many - or any - new pods on the nodes where those compute servers run, if doing so would overcommit the total CPU or memory available to Kubernetes for running pods on the node beyond the total cumulative amount of CPU or memory requested by all the containers in all the pods currently running on the node. So pre-starting compute pods effectively reserves a chunk of your total compute (or other pod-running) capacity, whether it is being used or not. TL;DR: The presence of pre-started compute server pods may prevent new compute pods from being started, even when those pre-started compute pods are idle. But with all that taken into consideration, if your users need to run a model pipeline - i.e. a series of model steps - in SAS Model Manager, or start SAS programs in Studio, and much quicker session startup time is worth the tradeoff of running the compute session as a shared account, then creating a pool of pre-started available compute servers might be for you!   Here's how.   Step 1: Store a credential for a shared service account   You must store a credential for a shared service account whether your identities are provided by LDAP or SCIM. However, I have only had a chance to try it with LDAP so far, so the steps below are specific to shared service accounts provided by LDAP. Consult Stuart's post for how to do the equivalent when your identities are provided by SCIM.   If you are using password authentication, and your users are provided by an LDAP provider such as Microsoft Active Directory (AD) or Azure Active Directory Services (AADS), read on.   First, authenticate to the sas-viya CLI as an administrator using e.g. sas-viya auth logon, or the pyviyatool loginviauthinfo.py (described in loginviauthinfo.md).   Then optionally, run this to list existing compute credentials:   sas-viya compute credentials list   If you have not created any compute credentials yet, the response will be:   There are no shared service account credentials.   Then run this to save a compute credential for a service account called modelservice , who has password mySecretPassword123 . Obviously, substitute the username and password of the service account your pre-started compute servers should actually run as for these example values:   sas-viya compute credentials create -u modelservice -p mySecretPassword123 -d "Shared service account called modelservice" The expected response is something like:   2023/06/07 12:34:56 The shared service account credential for modelservice was created successfully.   If you like, confirm it was created by listing compute credentials again:   sas-viya compute credentials list   If the credential was created successfully, the response will be something like:   Shared Service Account Credentials: 1. modelservice - compute-password - Shared service account called modelservice   Step  2: Create JSON file that defines the new compute context   The file below is based on the example in SAS Viya Platform Administration > Servers and Services > Programming Run-Time Environment > Programming Run-Time Servers > SAS Compute Server and Compute Service > CLI Examples. Save it somewhere which I will refer to as /path/to/file/prestarted_reusable_modelservice_cc.json .   { "name": "Data Mining compute context as modelservice", "description": "A compute context for the SAS Visual Data Mining and Machine Learning application as modelservice.", "attributes": { "runServerAs": "modelservice", "reuseServerProcesses": "true", "serverMinAvailable": "1" }, "launchContext": { "contextName": "Data Mining launcher context" }, "launchType": "service", "authorizedGroups": [ "Modellers" ] }   Adapt the JSON compute context template to your needs:   Change /path/to/file/prestarted_reusable_modelservice_cc.json to whatever file path and file name you wish to contain your JSON compute context definition Set our preferred name and description for this new compute context Provide the name of the service account you created in place of modelservice in the runServerAs attribute Optionally, you may wish to add further attributes: One that is quite useful is "serverInactiveTimeout": "900" . From the documentation reference in the next bullet: "Is used only when the reuseServerProcesses attribute is set. The value is the time in seconds that an inactive server remains running before being terminated. An inactive server is defined as a server without an active compute session.". When idle, you will see the pre-started compute servers terminate at the end of such an inactive period - and new ones are restarted if this brings the total of compute servers in the context below the number specified by serverMinAvailable . The Server Contexts section in SAS Help Center has a list of compute server attributes Set the value of the serverMinAvailable attribute to the initial number of compute servers you wish to have pre-started. See note below. Choose the most appropriate launcher context for the value of the contextName attribute: For SAS Model Studio use, leave it set to "Data Mining launcher context" For SAS Studio use, change it to "SAS Studio launcher context" Either change the group name in the authorizedGroups section from Modellers  to the groupID of the group you want to have access to this compute context, or if you do not want to restrict access to this compute context, remove the authorizedGroups section - and the comma at the end of  "launchType": "service" - to make this compute context available to all authenticated users.   Note: Remember that pre-started compute servers will consume some resources on your Kubernetes cluster's compute nodes, and as explained above will also reserve resources for potential use, so that they cannot be used for running other pods. The optimal serverMinAvailable number for the workload your users are running is something you may need to find by trial-and-error. I have seen a use case of a small environment with a couple of users who ran relatively small model pipelines, where 4 was a good value for serverMinAvailable . I have also seen another use case of a large shared classroom server used for teaching classes of students how to design models in SAS Model Studio, where 25 was was the chosen value for serverMinAvailable . This was the use case Rob used for the example values in his post. It is a value that you should tune, over time and with experience of what works best for your organization's workload.   Save your changes to /path/to/file/prestarted_reusable_modelservice_cc.json .   Step 3: Create the new compute context with pre-started compute servers   To create the new compute context, use the sas-viya command-line interface, still authenticated as an administrator, to run this command:   sas-viya compute contexts create -r -d @/path/to/file/prestarted_reusable_modelservice_cc.json   Note: the @ symbol in front of the file path is correct and required. You can of course create - and update, and delete - a new compute context interactively in SAS Environment Manager. My aim in this post is to show how to do it from the command line.   Step 4: See if compute servers have been pre-started   If you have access to a visual interface which shows pods running in your SAS Viya namespace in your cluster, you should see the number of sas-compute pods increase. However, depending on the interface, it may be difficult to distinguish the sas-compute pods started under this compute context (and running as the modelservice account) from other compute pods.   If you have access to the kubectl command-line interface for Kubernetes and a kube config file that lets you access your SAS Viya deployment's cluster, this command can show sas-compute pods launched specifically by the SAS Compute service (as opposed to by a user in SAS Model Studio or SAS Studio), and running as the modelservice account:   kubectl get pod -l launcher.sas.com/requested-by-client=sas.compute,launcher.sas.com/username=modelservice   Tip: Scroll the textbox above to the right if necessary, to see how the username who requested the pod is specified as a filter to kubectl get pods.   Here are some example results, when my new compute context's serverMinAvailable value was set to "1" , and I had no other SAS Compute sessions running in this compute context or as modelservice :   NAME READY STATUS RESTARTS AGE sas-compute-server-b91b1b9f-8827-4b57-9535-160cbf6e2cc4-t8mgl 1/1 Running 0 25s   Possible reasons you might not see the new compute servers starting up include:   You need to wait a little longer - a minute or two is usually enough The credentials for the service account are incorrect - try deleting and re-creating it There is not currently enough available CPU or memory to start new pods on Kubernetes nodes in your compute node pool - you may see fewer compute servers starting than you requested The account you specified in the runServerAs attribute is already running the maximum number of launched SAS Programming Run-time Environment sessions allowed, as set by the SAS_LAUNCHER_USER_PROCESS_LIMIT setting. See my blog post: Limit a user’s simultaneous compute server processes in Viya 2021.1 and later for more on this.   Assuming you did see the new sas-compute server pods starting up, you can now see if they improve performance of your model pipelines.   Use the new compute context in SAS Model Studio   Sign in to SAS Model Studio. If you limited access to the new compute context to only members of a specific group,  sign in to SAS Model Studio as a user who is in that group.   Then, create or open a model pipeline. To change the compute context used to run each step of this model pipeline, click the 'Settings' icon, in the top right corner of the project tab.   In the Project Settings dialog, choose Compute Context, and switch from the default 'Data Mining compute context' to your new compute context which pre-starts compute servers:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   I do not believe it is possible to configure individual steps in a model pipeline run under different compute contexts. The whole pipeline runs under the same compute context.   In the screenshot above, you can just about see a four-step sequential pipeline, created from a gradient boosting model in SAS Visual Analytics. When this project runs, each step runs in its own separate compute context. Some models run multiple steps at the same time. You should notice a significant improvement in the total 'wall clock' time (as compared to CPU time) when running these models while sufficient pre-started compute servers in the chosen compute context are available and not currently in use.   Use the new compute context in SAS Studio   SAS Studio users are generally familiar with switching compute contexts - click on the compute context icon top-right in SAS Studio to change to a different context:     If you select a compute context which has unused (i.e. available) pre-started compute servers, you should see a very noticeable improvement in how quickly your compute session in SAS Studio is ready to run SAS code.   The tradeoff for this is that the compute session runs as the shared service account, and not as you (or rather, as the signed-in user). This may have some impact on your filesystem permissions, if you access anything on the filesystem, such as base or other path-based libraries, filerefs etc. But the tradeoff may be worth it for the much faster compute session startup time.   So now you know how to script configuration of the compute context from the command line, and can see how easily it could be automated as part of an environment configuration script. Hopefully, that's useful to someone out there.   Thank you to Rob Collum for reviewing this post and for his great comments. Any remaining errors or omissions are mine.   See you next time!     Find more articles from SAS Global Enablement and Learning here. ... View more

I'm delighted to announce our new SAS Viya Administration Checklist is available now on GitHub:   https://github.com/sassoftware/viya4-admin-checklist   This checklist has 67 administration tasks for SAS Viya 2020.1 and later, organized into initial and regular task lists. There is a suggested frequency for the regular tasks. You, as an IT administrator or a SAS administrator, should consider performing the tasks in these lists for the SAS Viya 2020.1 and later environments that you maintain. Perform all tasks that are relevant to your environment to keep your SAS Viya deployment operating at its best over the long term.     This new checklist has been in development for a long time. The GEL Administration team and a select group of technical experts across the company have all worked on creating it, and we're so happy to make it available in this first release. But it is a living project, and we will add to it over time. If you see something you can improve, reach out to me or any of my GEL admin team colleagues, or raise a an issue in the GitHub project and let us know what you'd like to see changed, added or fixed. You can even contribute your own tasks. We have previously published similar checklists as PDF documents for earlier versions of SAS: Checklist of SAS Viya 3 Administration Tasks: https://support.sas.com/resources/papers/checklist-sas-viya-administration-tasks.pdf Checklist of SAS 9 Administration Tasks: https://support.sas.com/resources/papers/Platform-Administration-Tasks.pdf   Let me know what you think!   Acknowledgements   THANK YOU to all my SAS colleagues who contributed their suggestions and thoughts over the past several months. Not every suggestion made it into this release. We still have all the ideas that didn’t make this first cut, and will use those suggestions – and new ones – to drive future updates to the project. It will grow!   A particular thank you to my GEL Administration team colleagues: @ScottMcCauley, @GerryNelson ,  @AjmalFarzam and @GillesChrzaszcz who all wrote up tasks for this checklist. See you next time! ... View more

Someone contacted SAS Support recently asking for help with an issue in their SAS Viya platform running on Kubernetes. Support asked the customer to upload some logs from the relevant SAS Viya job to the support track. But this customer didn't have a way to get those logs. The job in question had ended, and the customer had no log (or metric) monitoring solution deployed or configured which had stored its log messages. The logs were gone.   This customer had no good way to access to their logs, except for the kubectl logs command which, in the case of transient pods like those which run compute jobs, is really only useful for pods that are still running. It's unlikely to be helpful for pods which have finished. So, basically, until this customer deploys some kind of log monitoring solution, and while they are at it a metric monitoring solution too, it will be near impossible for Support to help them troubleshoot that issue.   This is not the ideal first experience when you have just deployed the current SAS Viya platform. Please, if you are a SAS Administrator you need both a log monitoring solution and a metric monitoring solution for your SAS Viya platform.     Why do you need a log and metric monitoring solution for the SAS Viya platform?   In A first look at logging in SAS Viya 4 | Global Enablement & Learning Blog back in May 2020, I explained that the way SAS Viya [platform] log messages are collected and made available to SAS administrators is very different from the way they were handled in SAS Viya 3 or SAS 9. In correctly designed Kubernetes applications, the app should not attempt to write to or manage log files. Instead, log messages are simply written to an output stream: stdout and stderr . Kubernetes picks up this output and stores it in pod logs.   Rudimentary log viewing tools such as kubectl logs or a cloud service provider's Kubernetes web interface (which more or less offers the same functionality as kubectl logs in a web app) are good enough for simple use by e.g. a Kubernetes administrator and deployment consultants. But they don't offer long-term storage, aggregation over many pods, searching or much other functionality. In any case the SAS Administrators may not have access to them.   This means you need a log monitoring solution to capture SAS Viya log messages. The solution should store the log messages and their contextual information, and let you to chart, filter and search them.   While we're at it, you also need a metric monitoring solution, which will similarly capture and store metrics from your Kubernetes cluster and from your SAS Viya platform, store them, and let you chart, filter and explore them.   Plan to include both log and metric monitoring as part of your deployment. Add or enable both as soon as you can reasonably do so if you don't already have them! You may also want to ensure you have an alerting and at least a basic auditing capability. Collectively all these things fall under the category of observability tools.     Which log and metric monitoring tools should you use?   There are several choices. None is terribly expensive, or very difficult to deploy or use, but some are definitely better than the others.   Our top recommendation for observing SAS Viya deployments is SAS Viya Monitoring for Kubernetes. It consists of a log monitoring stack, and a metric monitoring stack, and a set of samples. It is documented in SAS Help Center, and is open source, supported by SAS, regularly updated by SAS developers and available license free on GitHub. It runs and is supported on all the Kubernetes and Cloud Service providers that SAS Viya is supported on.   If you are willing to do a bit more work or sacrifice a bit of convenience and data richness for the sake of consistency with the observability tool a customer already uses for other Kubernetes applications, all the major cloud service providers offer their own observability tools: Microsoft Azure and AKS has Azure Monitor, comprised of Azure Monitor Metrics and Azure Monitor Logs. Amazon AWS and EKS has Amazon CloudWatch, with Amazon CloudWatch Metrics and Amazon CloudWatch Logs. Google Cloud Platform and GKE has Google Operations Suite, with Cloud Logging and Cloud Monitoring. OpenShift OCP has the OpenShift Container Platform Monitoring Stack.   The SAS Viya Monitoring for Kubernetes project even has samples for how to configure each to work with SAS Viya.   If you are using Upstream Open Source Kubernetes to deploy SAS Viya (one of our newest options), you'd probably choose to create your hosting infrastructure and Kubernetes environment using SAS Viya 4 Infrastructure as Code (IaC) for Open Source Kubernetes, which works with SAS Viya 4 Deployment project on GitHub. This in turn can deploy SAS Viya Monitoring for Kubernetes with its cluster-monitoring and viya-monitoring tasks.   If you already have Kubernetes and/or already have another observability solution with log and metric monitoring support, you could use that.   Just be aware that if you choose something other than SAS Viya Monitoring for Kubernetes, some work is going to be involved to get the same richness of information from SAS Viya platform logs and metrics that is available in SAS Viya Monitoring for Kubernetes, and to keep it up-to-date over time as you update to new versions Kubernetes and the SAS Viya platform.   But please, do set up log and metric monitoring. It's essential.   Further reading Learn more by reading my first look at logging in SAS Viya 4.   To see more recent versions of SAS Viya Monitoring for Kubernetes in action, watch: SAS Demo | Viewing Logs with SAS Viya Monitoring for Kubernetes: Changing Log Levels - YouTube. ...and its supporting SAS Communities post: Code for YouTube video: "Viewing Logs with SAS Viya Monitoring for Kubernetes: Changing Log Levels SAS Demo | Viewing Logs with SAS Viya Monitoring for Kubernetes: Managing Log Retention - YouTube Look for other posts on observability, logging and monitoring (or log monitoring and metric monitoring) here, including: Four Tips for Exporting logs from OpenSearch Dashboards’ Generate CSV function Managing logging data retention period and size in SAS Viya Monitoring for Kubernetes Managing monitoring data retention period and size in SAS Viya Monitoring for Kubernetes Tips for upgrading SAS Viya Monitoring for Kubernetes 1.1.8 to version 1.2.0 Log monitoring updates in the SAS Viya Monitoring for Kubernetes project Overview of updates to the SAS Viya Monitoring for Kubernetes project Exporting logs from SAS Viya on Kubernetes – a survey of five available methods Managing access to logs for specific Viya deployments with Kibana A second look at Azure Monitor Logs for AKS clusters A first look at Azure Monitor Logs for AKS clusters   Find more articles from SAS Global Enablement and Learning here. ... View more

SAS Watchdog is an optional component for the SAS Viya platform's Programming Run-time, which architects and administrators should know about. It has been documented in the READMEs and HTML doc that come with your SAS Viya  platform deployment assets for a long time now, but in the Stable 2023.01 release of the SAS Viya platform, that doc has been improved and SAS Watchdog is covered in the Help Center documentation too.   If you enable SAS Watchdog, it helps keep customer environments secure for users who run their own open source code written in Python, R or Java from within a SAS Compute, Connect or Batch programming session.   In this post, I want to tell you what SAS Watchdog is, how it works, and why you would choose to enable it (or not). This is partly just because it's really clever, but also because SAS Watchdog is an essential tool that can keeping your customer's data safe when they run code written in other programming languages from within a SAS program.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Most importantly, sas-watchdog restricts access to the filesystem using the same allowlist that SAS's LOCKDOWN feature uses. In a future post, we'll look at how to manage both the SAS Programming Run-time allowlist that is relevant here, and also the CAS allowlist, which is a seperate list but which implements a similar concept for SAS CAS Language processing.   Why you need a way to restrict filesystem access from the SAS Programming Run-time   SAS Watchdog is needed because of the way two features of the SAS programming language need to work together: LOCKDOWN, and SAS's ability to run user-written code in other languages.     LOCKDOWN prevents unauthorized access to the filesystem from the SAS programming language   Native SAS code running in a compute, connect or batch programming session is prevented from accessing filesystem paths which should be off-limits by the LOCKDOWN feature of the SAS programming language. It also limits access to certain language features and access methods, but in this post we are more concerned with its function of limiting filesystem access. After initialization, any attempt to access the filesystem from any SAS language feature - a procedure, data step or path-based library - will be checked against an allowlist of filesystem paths by the SAS programming run-time.     If the directory or file is in or below a path in the allowlist, it is allowed. If the directory or file is not in the allowlist and access to it is attempted after the programming session enters its locked-down state, the access is prevented.   There is an exception to this: autoexec code which is managed by SAS administrators, and is processed during the compute (or connect or batch) server/process initialization can define file references and path-based library references to paths outside the allowlist. These references have to be set up during initialization, before the SAS programming session enters its locked-down state. By the time end-user-written code can begin to run, the SAS programming session has already entered a locked-down state and no new file or library references to paths outside the allowlist are allowed.     The SAS Programming Run-time supports calling other programming languages   One of the most exciting features of the SAS programming run-time is that it supports running user-written code in other languages: Python, R and Java.   @ScottMcCauley wrote two great posts recently about how to Configure Python Integration with SAS Viya. and Configuring R Integration with SAS Viya. Together, the two posts (and the documentation they link to) show how the SAS Configurator for Open Source and other steps can be used to enable SAS programs to run Python (e.g. via proc python; ), and R (e.g. via proc IML; ). SAS has also long supported running Java code via the data step Java Object.     When code in another language is executed, the compute (or connect or batch) server uses another SAS process called sasels to launch processes running the respective programming language's own run-time binaries as the user who started the SAS programming session. It passes the code to be executed (or a filepath pointing to a file containing the code) to the respective programming language's executable.   Of course, the Python, R and Java languages are all fully-featured. They also have features which allow them to access the filesystem. And obviously those language runtimes do not have the SAS programming run-time's LOCKDOWN feature. Plus, the top-level processes called when another language's code is executed are also capable of launching additional processes themselves.   So, what will prevent a user running a proc python; statement which executes Python code to read a file outside the SAS Programming Run-time's allowlist? Equally, the same goes for R and Java code.   Sure, the only file system volumes that it has access to, since it is running inside a Kubernetes pod are the paths mounted into the pod. The container runtime (Docker, containerd etc.) and Kubernetes together prevent a process from accessing the host file system, other pods, other hosts etc. But still, it is not desirable to give a launched processes running arbitrary user-written code access to all filesystem paths mounted into the pod!     SAS Watchdog   SAS Watchdog's job - if it is enabled - is to prevent user-written code in one of those languages, started inside a compute (or connect or batch) programming session from exploring the compute (or connect or batch) pod's filesystem, and from accessing files mounted to it which they have no business accessing.   Once you enable it, SAS Watchdog runs as a process inside its own separate sidecar container, in each SAS Programming Run-time pod. In Kubernetes a 'sidecar container' is an additional container in a pod (beside the main container), which stays running after a pod's init containers have finished their initialization.   However, SAS Watchdog is disabled by default.     How SAS Watchdog works     Follow the numbered annotations in the diagram above and the list below to see how SAS Watchdog works. If it is enabled, SAS Watchdog runs with elevated privileges that allow it to register itself with a LINUX kernel API called fanotify, part of the operating system running inside the pod. Once SAS Watchdog has registered itself with fanotify, any request to access the filesystem - such as a file open request - coming from any process launched by the sasels process will trigger an 'open event' notification which will be sent to SAS Watchdog. When a process launched by sasels - including any Python, R or Java processes, and also any sub-processes they spawn - requests access to the filesystem, fanotify sees that a file open request has been made by a watched process. The LINUX kernel does not grant or deny the request yet. Fanotify sends an event to the SAS Watchdog process, and waits for it to respond. The SAS Watchdog process consults the list of filesystem paths in the allowlist, and determines whether or not the requested file is in or below a path on the allowlist. If the file or directory requested is in the allowlist, SAS Watchdog responds to the open event to indicate that it allows the request. If not, it responds saying it denies the request. If SAS Watchdog allowed the request, the LINUX kernel grants the file open request and the calling process gets access to the file or directory. If SAS Watchdog denies the request, the kernel denies that request to access the file.   The allowlist that watchdog uses is the exact same one that LOCKDOWN uses. Inside the pod, it's read from the same file by both processes. So if you change the list of paths in the LOCKDOWN paths list, there is nothing else you need to do to have your change affect the SAS Watchdog process.     How can you enable SAS Watchdog   SAS Watchdog is disabled by default, but can be enabled in your SAS Viya platform deployment in the same way as so many other SAS Viya platform features, with an overlay you copy to your site-config directory, and reference from your kustomization.yaml, so that (along with many other overlays) it will be used by kustomize to modify your site.yaml file before site.yaml is applied to your Kubernetes cluster, following whichever SAS Viya platform deployment method you use at your site.   Specifically, read the instructions in:   the README file at $deploy/sas-bases/overlays/sas-programming-environment/watchdog/README.md (for Markdown format) or the web page at $deploy/sas-bases/docs/configuring_sas_compute_server_to_use_sas_watchdog.htm (for HTML format)   Using the instructions that are shipped with the deployment assets in your specific release of the SAS Viya platform ensures you have the right instructions for that specific release. But the general gist of those instructions is that you add a reference to the  sas-programming-environment/watchdog  overlay to the transformers block of the base kustomization.yaml file ( $deploy/kustomization.yaml ), like this   ... transformers: ... - sas-bases/overlays/sas-programming-environment/watchdog - sas-bases/overlays/required/transformers.yaml ...   NOTE: The reference to the  sas-programming-environment/watchdog  overlay MUST come before the required transformers.yaml, as seen in the example above.   If you are running the SAS Viya platform on an OpenShift environment, there are additional steps in the instructions for applying a Security Context Constraint that is required for SAS Watchdog to work.   Check your release's instructions, at the paths given above, to be sure you are following the right process for your release. These instructions were updated and refreshed for SAS Viya platform Stable cadence release 2023.01.   And take a look at the new documentation for SAS Watchdog in the SAS Viya Platform Administration Guide.     Why might you choose not to enable SAS Watchdog?   The SAS Watchdog process in its sidecar pod has to run with elevated privileges in order to register itself with the fanotify API. Some Kubernetes administrators, or their parent organization may not approve of this. Discuss this with your architect and customer during the architecture phase of an implementation project.   If you disable access to the JAVA Data Step Object using LOCKDOWN, and do not enable Python and R code to be run from within SAS Programming Run-time servers (SAS Compute, SAS/CONNECT and SAS Batch), there is no need to enable SAS Watchdog, and doing so would consume a little bit of system resource for no additional benefit.   Besides those two reasons, I cannot immediately think of any reason why you would not enable SAS Watchdog. But feel free to contact me or add a comment below if you can!   See you next time!   Find more articles from SAS Global Enablement and Learning here. ... View more

I love it when people ask our team questions. I love it even more when we can answer them, and a few days ago a very knowledgeable colleague asked me a really good question:   Is there a way to set a permission to none using the sas-viya CLI like you can do in the GUI?   It helps enormously if you use a couple of additional command-line tools (jq and the getruleid.py pyviyatool) in addition to the sas-viya CLI, but with their assistance, yes, you can set a permission to '(none)' using the sas-viya CLI.   This is what my colleague was asking about:   Setting a permission to none using the SAS Environment Manager GUI   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Read on to learn how to do this from the command line. It's not quite as simple as it looks!   Demo Preparation   To set up a folder to work with, I used to an instance of the SAS Viya environment that we provide for students attending our team's GEL SAS Viya: Administration workshops. These workshops have a set of fictitious users who are in groups, and who work for a fictitious company called GELCorp. There is a Managers group ready made with a couple of users in it. We have a ready-made folder structure in these environments, including a Viya folder called /gelcontent/GELCorp.   To prepare, I ran some sas-viya CLI statements to create a 'Managers' Viya folder under /gelcontent/GELCorp, and to grant the Managers groups some read-write permissions on that folder (targeted by --object-uri ) and on its contents (targeted by --container-uri) :   # Create a Managers folder in /gelcontent/GELCorp /opt/sas/viya/home/bin/sas-viya --output text folders create --parent-path /gelcontent/GELCorp --name Managers #Get /gelcontent/GELCorp/Managers folder id, store in $folderid folderid=$(/opt/sas/viya/home/bin/sas-viya --output json folders show --path "/gelcontent/GELCorp/Managers" | jq -r '.["id"]') echo "The Managers folder ID is" $folderid # Grant Managers group: +RUARe,+RUDARe& echo "Granting Managers +RUARe,+RUDARe& on /gelcontent/GELCorp/Managers" /opt/sas/viya/home/bin/sas-viya authorization grant --object-uri /folders/folders/$folderid/** --group Managers --permissions read,update,add,remove /opt/sas/viya/home/bin/sas-viya authorization grant --container-uri /folders/folders/$folderid --group Managers --permissions read,update,delete,add,remove   Here's the command line output from that series of commands:   [cloud-user@hostname ~]$ # Create a Managers folder in /gelcontent/GELCorp [cloud-user@hostname ~]$ /opt/sas/viya/home/bin/sas-viya --output text folders create --parent-path /gelcontent/GELCorp --name Managers Id 84a65308-b77f-4844-814c-02fcc51221ba Name Managers Description Type folder MemberCount 0 ParentFolderUri /folders/folders/ce8f1f43-3af4-4cc9-a360-922534670eec The folder was created successfully. [cloud-user@hostname ~]$ [cloud-user@hostname ~]$ #Get /gelcontent/GELCorp/Managers folder id, store in $folderid [cloud-user@hostname ~]$ folderid=$(/opt/sas/viya/home/bin/sas-viya --output json folders show --path "/gelcontent/GELCorp/Managers" | jq -r '.["id"]') [cloud-user@hostname ~]$ echo "The Managers folder ID is" $folderid The Managers folder ID is 84a65308-b77f-4844-814c-02fcc51221ba [cloud-user@hostname ~]$ [cloud-user@hostname ~]$ # Grant Managers group: +RUARe,+RUDARe& [cloud-user@hostname ~]$ echo "Granting Managers +RUARe,+RUDARe& on /gelcontent/GELCorp/Managers" Granting Managers +RUARe,+RUDARe& on /gelcontent/GELCorp/Managers [cloud-user@hostname ~]$ /opt/sas/viya/home/bin/sas-viya authorization grant --object-uri /folders/folders/$folderid/** --group Managers --permissions read,update,add,remove { "acceptItemType": "", "acceptType": "", "contentType": "", "createdBy": "geladm", "creationTimestamp": "2022-12-15T16:35:38.835Z", "description": "", "enabled": true, "id": "9f585603-f29b-445a-8e19-1aba538fae8b", "mediaType": "", "modifiedBy": "geladm", "modifiedTimestamp": "2022-12-15T16:35:38.835Z", "objectUri": "/folders/folders/84a65308-b77f-4844-814c-02fcc51221ba/**", "permissions": [ "remove", "read", "update", "add" ], "principal": "Managers", "principalType": "group", "reason": "", "type": "grant", "version": 10 } The authorization rule has been created. [cloud-user@hostname ~]$ /opt/sas/viya/home/bin/sas-viya authorization grant --container-uri /folders/folders/$folderid --group Managers --permissions read,update,delete,add,remove { "acceptItemType": "", "acceptType": "", "containerUri": "/folders/folders/84a65308-b77f-4844-814c-02fcc51221ba", "contentType": "", "createdBy": "geladm", "creationTimestamp": "2022-12-15T16:36:23.113Z", "description": "", "enabled": true, "id": "c626a6de-60dd-4798-a928-a2568f5de466", "mediaType": "", "modifiedBy": "geladm", "modifiedTimestamp": "2022-12-15T16:36:23.113Z", "objectUri": "", "permissions": [ "remove", "read", "delete", "update", "add" ], "principal": "Managers", "principalType": "group", "reason": "", "type": "grant", "version": 10 } The authorization rule has been created. [cloud-user@hostname ~]$   In SAS Environment Manager, in the Content page, the Managers group's permissions on the new folder (right-click > View Authorization) look like this:     Initial permissions set on the new Managers folder   Note that among other permissions, I have granted the Managers group 'Update' permission on the Managers folder itself.     Set a permission to none using the sas-viya CLI   Let's suppose a few days have passed, and I notice that Managers (such as the user 'Helena') in our workshop environment have permission to rename the Managers folder!   Helena can rename the Managers folder!   Oops! That was not my intention.   Helena can rename the Managers folder because I 'accidentally' granted Managers (a group she belongs to) the Update permission on the folder (she already had Update permission on its parent folder, /gelcontent/GELCorp).   I want to remove that grant - in other words, to set the permission to '(none)'. It's easy to do in the SAS Environment Manager GUI's Edit Authorization dialog, but how can we do it using the sas-viya CLI? There isn't exactly a sas-viya CLI syntax that will set a permission to '(none)'.   What we need to do is update the rule that grants the read, update, add and remove permissions on the Managers folder to members of the Managers group, so that it only grants read, add and remove permissions - and not update any more.   We can do this with the sas-viya authorization update-rule command. But the catch is that it requires a rule ID, which I don't know. The commands that created the new general authorization rules that represent the permissions I granted did not include the rule IDs in their output, and in any case, let's assume some time passed since I created it and I wouldn't remember them even if the rule IDs were output at the time. And short of looking up the rule in SAS Environment Manager, which defeats the purpose of using the sas-viya CLI, we need a way to find the rule ID using CLI commands.   I spent a while trying to do this using the sas-viya CLI, with the sas-viya authorization list-rules command and jq. I think it should be possible. However, after a while, I remembered I wrote a getruleid.py pyviyatool for just this sort of purpose! My colleague @GerryNelson  wrote a blog post explaining how to install and use the pyviyatools in 2018, but it's still correct and relevant, and there are more useful tools in the set now too!   That made it much easier, as getruleid.py only requires a folder ID and a identity principal (a user or a group), and returns all the details of the rule - including its id - in JSON, CSV or a simple multiline name-value format.   So, the following code will do the trick - first, it uses the sas-viya CLI and jq to get the folder ID. Next, it uses getruleid.py and awk to get the rule ID. Finally, it uses the sas-viya authorization update-rule command to change the permissions that the rule grants, reducing them from 'read,update,add,remove' to just 'read,add,remove', which effectively sets the 'update' permission to '(none)'.   #Get /gelcontent/GELCorp/Managers folder id, store in $folderid folderid=$(/opt/sas/viya/home/bin/sas-viya --output json folders show --path "/gelcontent/GELCorp/Managers" | jq -r '.["id"]') echo "The Managers folder ID is" $folderid # Get the id of the rule targeting that folder for Managers, and parse the output a bit to keep just the rule ID ruleid=`/opt/pyviyatools/getruleid.py -u /folders/folders/$folderid/** -p "Managers" -o simple | grep "id = " | awk '{print $3}'` echo "The rule ID is" $ruleid # Reduce the permissions that Managers have on the folder, keeping just 'Read', 'Add' and 'Remove', which effectively removes the unwanted 'Update' permission: /opt/sas/viya/home/bin/sas-viya authorization update-rule --id $ruleid --grant --group "Managers" --permissions read,remove,add   Here's the command line output from that series of commands:   [cloud-user@hostname ~]$ #Get /gelcontent/GELCorp/Managers folder id, store in $folderid [cloud-user@hostname ~]$ folderid=$(/opt/sas/viya/home/bin/sas-viya --output json folders show --path "/gelcontent/GELCorp/Managers" | jq -r '.["id"]') [cloud-user@hostname ~]$ echo "The Managers folder ID is" $folderid The Managers folder ID is 84a65308-b77f-4844-814c-02fcc51221ba [cloud-user@hostname ~]$ [cloud-user@hostname ~]$ # Get the id of the rule targeting that folder for Managers, and parse the output a bit to keep just the rule ID [cloud-user@hostname ~]$ ruleid=`/opt/pyviyatools/getruleid.py -u /folders/folders/$folderid/** -p "Managers" -o simple | grep "id = " | awk '{print $3}'` [cloud-user@hostname ~]$ echo "The rule ID is" $ruleid The rule ID is 9f585603-f29b-445a-8e19-1aba538fae8b [cloud-user@hostname ~]$ [cloud-user@hostname ~]$ # Reduce the permissions that Managers have on the folder, keeping just 'Read', 'Add' and 'Remove', which effectively removes the unwanted 'Update' permission: [cloud-user@hostname ~]$ /opt/sas/viya/home/bin/sas-viya authorization update-rule --id $ruleid --grant --group "Managers" --permissions read,remove,add { "acceptItemType": "", "acceptType": "", "contentType": "", "description": "", "enabled": true, "id": "9f585603-f29b-445a-8e19-1aba538fae8b", "mediaType": "", "modifiedBy": "geladm", "modifiedTimestamp": "2022-12-15T17:19:16.495Z", "objectUri": "/folders/folders/84a65308-b77f-4844-814c-02fcc51221ba/**", "permissions": [ "remove", "read", "add" ], "principal": "Managers", "principalType": "group", "reason": "", "type": "grant", "version": 10 } The rule has been modified. [cloud-user@hostname ~]$   If you look at the command output closely (and scroll down), you can see that this rule now only grants the smaller set of permissions we asked for. Here are how the Managers group's permissions on the Managers folder look after this correction:      Corrected Managers folder permissions   The Managers group no longer has the 'Update' permission granted directly on the Managers folder. Since nothing else grants Managers this permission, the effective permission decision for the 'Update' permission for 'Managers' on the Managers folder is 'denied'. In the absence of an grant, permissions in the SAS Viya general authorization system are denied by default.   As a result, the Helena can no longer rename the Managers folder:    Helena can no longer rename Managers folder   Job done. My thanks to my colleaque Bruno Muller for asking a really good question, with a not-so-straightforward answer. And I hope you will find this useful in your projects!   See you next time!   ... View more

In release 1.2.0 of SAS Viya Monitoring for Kubernetes in June 2022, we got a great new way to export SAS Viya log messages to a file, the Reporting > Generate CSV function in OpenSearch Dashboards:   Reporting > Generate CSV menu button   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   At the risk of sounding like click-bait, with four simple tips it's easy to export the SAS Viya log messages you're interested in. You can include just the fields of contextual information you want, and your exported messages can span multiple services over multiple days. These tips will help you work around a bit of quirky behavior, and get the most out of this excellent new(-ish) feature.   As a brief aside while on the topic of exporting logs from SAS Viya, I last wrote about it in this post in April 2021. A couple of things have evolved about the methods I surveyed back then:   Mirantis Lens is still great, but now requires you to create an account and sign in before you can use it. A near equivalent project called OpenLens does not require you to have an account or sign in. You either have to download and trust their unsigned binary installation files, or you have to clone the source code and build/make it yourself. But we like it because in our workshops, not having the attendees need to create an account and sign in removes a little bit of friction. The experimental getlogs.sh script in the SAS Viya Monitoring for Kubernetes project has not received much maintenance in the past year or so, and as a result some key features, like searching and filtering, no longer seem to work. When I last tried using the getlogs.sh script with search or filtering parameters in release 1.2.6, it returned no results when I think it should have done. It always was a little quirky, and was always clearly categorized as EXPERIMENTAL, so I doubt this effects many users. Given the range of good alternatives available and competing development priorities for the team who maintain it, I don't expect to see the getlogs.sh utility become production ready any time soon. You should plan to use something else for the short to medium term.   Four Tips for using the Generate CSV function   Now, on to the main subject of this post: the Reporting > Generate CSV function in OpenSearch Dashboards' Discover page. This will export log messages as seen in the table below the chart on the discover page.   You need to know four things to get best results:   The Time column in the table is not exported. I don't know why not, but its omission from CSV exports appears to be a deliberate choice by the developers. You should add the @timestamp column to the table if you want a timestamp in your CSV export - and why wouldn't you? You likely prefer to move the @timestamp column to the left of the CSV results. But the Time column (which is not exported) is fixed as the first column on the left side of the table. So, you will likely prefer to move the @timestamp column left as far as it will go, which will be the second column on the left side of the table, one to the right of the Time column. OpenSearch generates the CSV results based on the filter and search criteria, and the table columns in a saved search, not based on what you currently see on screen. This can be unintuitive. Changes you make to the Discover page will have no effect on the content of the exported CSV unless you first save them as a saved search. Tip: you should not overwrite the 'Log Messages' saved search. It is provided with your SAS Viya Monitoring for Kubernetes logging stack during its deployment. This saved search is used in a number of dashboards, and changing it (e.g. by adding a filter or search) will impact dashboards based on it. Because dashboards display charts, the effect of changes to the 'Log Messages' saved search may not be immediately obvious. Leave 'Log Messages' alone and save your search and filter criteria as a new saved search before exporting log data to a CSV with the Reporting > Generate CSV function. If you are unable to generate a CSV as the logadm user, generate one as the admin user, just once. Afterwards, the logadm user should be able to generate CSVs.   Let's see why each of these things matters.     Tip 1: Time column not exported; add @timestamp column   Here's a simple saved search I created by opening the Log Messages saved search, adding a filter for 'logsource = cas', and saving it as a new saved search named 'Log Messages - CAS':    On the Discover page of OpenSearch Dashboards, the leftmost column of the table of log documents is always Time   When I choose Reporting > Generate CSV, the resulting CSV data does not include the Time column, and have not found any option that you can change to include it:   The Time column is not included in a generated CSV based on the same saved search   The solution is to add the @timestamp column to the table in your saved search...   Add @timestamp column to the table   ...then remember to save it and lastly export the CSV again:   Now the CSV has a timestamp column… wherever it was in the table. Here it is the right-most column, which looks a bit strange.   Tip 2: Move @timestamp column as far left as it can go, to the second column from the left It's nice to have the @timestamp column first in the CSV. To achieve this, click the little double left chevrons ('«') in the column heading for the @timstamp column - they appear when you hover your mouse pointer over them:    Left double chevrons (‘«’) in the @timestamp column heading. Click to move column to the left.   Keep doing this until the @timestamp column is as far to the left as it will go. This is the second-left most column. The 'Time' column which is not exported in the CSV, is always the left-most column:   The timestamp column moved to the left as far as it will go   Remember again to save your changes to your saved search, and choose Reporting > Generate CSV again. The @timestamp column will now be at the left of the CSV:   CSV with @timestamp column on the left Tip 3: Save the filter and query state of the Discover page as a Saved Search before Generating a CSV   In the tips above, I reminded you to save your changes (but don't overwrite the out-of-the-box Log Messages saved search) before exporting a CSV. There are two reasons for this.   First, if you just open the Discover page in OpenSearch Dashboards, without opening or saving your view as a saved search, you will see that the Reporting > Generate CSV function is disabled:    No saved search is open in the Discover page. The Reporting > Generate CSV option is disabled.   Second, after you change any of the filters, search string, columns, sort order or anything else about the Discover page, if you do not then save your changes to update your saved search, and choose Reporting > Generate a CSV, the CSV will reflect the filters, search string, columns, sort order etc. as they were when you last saved them to the saved search object, and will NOT reflect your changes.   If you try to generate a CSV without saving recent changes, there is no prompt asking if you want to save your changes. This is not great UI design, but it's not too difficult to get into the habit of first saving, and then generating a CSV.     Tip 4: Generate a CSV as the admin user first before trying to do it as the logadm user   Some initialization happens behind the scenes in OpenSearch the first time you generate a CSV this way.   SAS Viya Monitoring for Kubernetes configures a 'non administrator' user in OpenSearch called logadm, which you can use when you don't need full administrator privileges, to reduce the risk of you making an accidental configuration change.   It seems that in SAS Viya Monitoring for Kubernetes prior to version 1.2.7  the admin user has all the permissions it requires to complete the initialization steps for generating a CSV, while the logadm user does not.   So, if you first try to generate a CSV in a new deployment of the SAS Viya Monitoring for Kubernetes logging stack as logadm, it fails like this:   Generating a CSV as the logadm user first fails. Generate one as admin first.   The error message says "Error generating report", "Insufficient permissions. Reach out to your OpenSearch Dashboards Administrator".   The solution is to sign in as the admin user, and generate a CSV from any saved search.  This should succeed, and without you seeing any evidence of it the initialization steps are performed successfully. Then sign out and sign in again as the logadm user. Now generating CSVs as logadm should succeed too.     I hope these tips have been helpful. Please let me know in the comments below if you have any more tips for generating CSVs, or for any other feature in SAS Viya Monitoring for Kubernetes' logging stack (OpenSearch and OpenSearch Dashboards).     See you next time!   Find more articles from SAS Global Enablement and Learning here. ... View more

There are at least four different breadths of scope over which you can configure the SAS Programming Run-time. They are:   SAS Launcher: broadest scope, affects all Programming Run-time sessions in the whole SAS Viya deployment Compute, Connect or Batch Service Compute, Connect or Batch Context User-specific autoexec, user-specific custom SAS code: narrowest scope, affects only one user, in SAS Studio   Not all of these breadths of scope apply to all services or clients. Even so, this is really useful because it allows, e.g. the maximum number of SAS Programming Run-time sessions a user is allowed to have running at once to be defined in a single place and yet apply across every SAS programming run-time sessions in the whole SAS Viya deployment. At the same time, other settings e.g. a very task-specific SAS macro or library definition can be defined only for the user who needs it, or only in the programming run-time context in which it is used. The intermediate locations provide degrees of broader or narrower scope between those extremes.   All this gives SAS Administrators fine control over how broadly or narrowly individual aspects of programming run-time session, such as localization (language, character encoding, timezone), LOCKDOWN paths, macro definitions, SAS options and other configuration are applied. It's good for SAS Administrators to know all the places where they can apply such configuration, to affect just the programming run-time sessions they intend to affect.   So, in this post, we'll review the main parts of the SAS Programming Run-time that can be configured, and we'll consider the effective breadth of scope - which sessions they will affect - for their configuration settings.         SAS Launcher   In SAS Viya, all compute and batch sessions, and by default all connect* sessions, run on processes in sas-programming-environment containers in pods which are started by the SAS Launcher Service.   What can you change?   Examples of global configuration settings which can be adjusted for Kubernetes resource definitions at the SAS Launcher level include: Modifications to all podTemplates used by SAS Launcher, targeted like this: "target": {"labelSelector": "sas.com/template-intent=sas-launcher", "kind": "PodTemplate"} . To pick an example, you might change the path of the viya volumes in these podTemplates, to relocate temporary files such as those used in the SASWORK library to faster or larger storage than whatever is used by a Kubernetes emptyDir in your cluster. Adjustments to the amount of memory and CPU made available to launched SAS Programming Run-time pods. These adjustments are made by setting one or more of a set of eight variables that control default and maximum values for the CPU and Memory requests and limits in launched pods. When SAS Workload Management is not included in the deployment, changes to the SAS_LAUNCHER_USER_PROCESS_LIMIT, which sets an upper limit to the number of launched pods that each user can have running at once. This limit applies across launched compute, connect and batch pods. When SAS Workload Management is included in the deployment it takes over responsibility for launching all compute, connect and batch servers from the SAS Launcher. In this case, as my colleague Rob Collum wrote in his post Improving Concurrency Performance in SAS Viya, you must use a alternative setting to limit Maximum Jobs Allowed per user: As an administrative user in SAS Environment Manager, navigate to the SAS Workload Orchestrator configuration Under Host Types, expand the desired section (“default” is the default) Change the maximum number of jobs value to a better value, like 50, or 100. There is no global autoexec.sas code block or file. To globally define the sorts of things that go in an autoexec, like libraries and macros or SAS Options, you must add the definition separately to each of the Compute, Connect and Batch service autoexec_code blocks.   Where can you change it?   Changes to these things are best made in YAML files in your SAS Viya deployment's site_config directory, by copying and editing sample overlays. There are some additional examples here: SAS Help Center: Common Customizations.   What is the scope of these changes?   Configuration changes applied to the SAS Launcher Service have a global scope - they affect every launched* SAS Programming Run-time server, and the session(s) it runs. * SAS/CONNECT in older versions of SAS Viya could spawn connect sessions directly in the SAS Connect Service pod, without launching any new pods. This can still be enabled, but in recent releases spawned connect sessions are disabled by default.       Compute, Connect and Batch Services   The SAS Viya Compute, Connect and Batch Services run all the time. Each of these acts as a front end for client applications and services to launch, submit code to, check the status of and get program logs and results from programming sessions.   What can you change?   Each of them has configuration settings which allow you to specify autoexec code that runs while a compute, connect or batch pod initializes its respective programming session. You can also specify configuration options (SAS Options like in a sasv9.cfg file in SAS 9), restricted options whose values are set by the SAS Administrator and cannot then be overridden by users, and startup commands for each of the three kinds of SAS Programming Run-time server. The full set is documented here in the SAS Viya Administration Guide.   Where can you change it? Configuration for the Compute, Connect and Batch services is stored in the SAS Configuration Server. View and modify it in SAS Environment Manager's Configuration page, or using the sas-viya CLI's configuration plug-in, and its configurations keyword, e.g. (using jq to help): sas-viya configuration configurations show --id $(sas-viya configuration configurations list --definition-name sas.compute.server | jq -r '.items[] | select (.name | contains("autoexec_code")) | .["id"]') What is the scope of these changes?   The configuration options for each type of SAS Programming Run-time server apply to all programming sessions started under that type of server, and are specific to that type of server; they don't impact the other two types.       Launcher Contexts   Per the documentation, A launcher context is a specification that enables SAS administrators to apply environmental and access constraints on processes that are run by a launcher service. Each Compute, Connect and Batch context references a single Launcher context. The Launcher context references a podTemplate.   What can you change?   Launcher Contexts represent some - though not all - of the configuration to be applied when starting compute, connect and batch servers.   They specify which podTemplate should be used to create the compute, connect or batch server pod. Each podTemplate comes with a number of configuration settings of its own.   They also provide a place to define environment variables, e.g. to set character encoding or locale, specify OS commands - usually shell scripts - which must be executed as the SAS programming session starts, and/or specify the queue that the session should be placed in for SAS Workload Orchestrator. Personally, I have not come across a use case for modifying a launcher context yet, but I have not needed to change a deployment's character encoding and locale or configure its Workload Management queues lately. This is a good place do those things.   Where can you change it?   Launcher Contexts are managed by the Launcher service, and stored in the SAS Configuration Server.     View and modify them in SAS Environment Manager's Contexts page (in the Launcher Contexts view), or using the sas-viya CLI's compute plug-in, and its launchers keyword, e.g. sas-viya compute launchers list .   The podTemplates referenced by launcher contexts can also be modified (like the SAS Launcher service itself) using YAML files in your SAS Viya deployment's site_config directory, by copying and editing sample overlays.   What is the scope of these changes?   Changes to a launcher context affect SAS Programming Run-time sessions started in a compute, connect or batch context that use that launcher context. You can usually tell which these are from the launcher context name. But you may need to inspect a number of compute, connect or batch contexts to find all of them that use a specific launcher context.       Compute, Connect and Batch Contexts   Compute, Connect and Batch Contexts provide SAS Administrators with a place to define configuration settings for SAS Programming Run-time sessions. Documentation: SAS Help Center: Server Contexts: Concepts.   Each context is intended to be used by a either specific SAS client (e.g. SAS Studio, SAS Model Manager), or for a specific purpose (e.g. data mining, managing CAS formats, running backup jobs), and you can even create compute contexts for specific groups of users.   In some client applications like SAS Studio, the end user can choose which context their programming session runs under, from a list that is filtered to only show contexts the user has permission to use. Out of the box, all compute, connect and batch contexts are accessible by all users, but a SAS Administrator can create their own contexts (of any of the three types), and can restrict which users and groups can use them.   What can you change?   Several compute contexts are created out-of-the-box in a SAS Viya deployment, and you can add your own. Out-of-the-box (in my deployments) there is only one connect context and two batch contexts. As my colleague Edoardo Riva pointed out, starting in SAS Viya 2021.2.6  (released in April 2022), there is a second batch context for command-line commands. If your deployment of SAS Viya is an earlier version than 2021.2.6, you will likely only have one batch context out-of-the-box, but you can create your own.   Compute contexts (and by extension, presumably batch and connect contexts too), provide a place to specify attributes that modify the behaviour of a compute (or connect or batch) session. Examples include runServerAs, used to run sessions under the context as another user, reuseServerProcesses used to make compute servers reusable, or serverMinAvailable, used to pre-start a pool of available servers.   They also have a place to define additional SAS autoexec statements, SAS options or a SAS Workload Management queue.   And, through their associated launcher context, they provide an indirect way to use a different podTemplate from the default, if you have created additional podTemplates.   Finally, a new feature in SAS Viya 2022.1.3 and later allows SAS Administrators to define Library Connections (also called 'Connections to Data Sources' or 'Data Source Definitions') with the 'Allow all users to view the library connection' option selected. If this option is selected for a Library Connection resource, it is associated with the current compute context as a 'Resource', and will be declared for all compute sessions started under that compute context. See @GerryNelson's post: Managing Connections to Data Sources for Compute sessions in SAS Viya.   Where can you change it?   Compute, Connect and Batch Contexts are managed by their respective services, and stored in the SAS Configuration Server. View and modify them in SAS Environment Manager's Contexts page (in the Compute contexts, Batch contexts and Connect contexts views).   You can also modify compute and batch contexts using the corresponding plugin ( compute or batch ) for the sas-viya CLI. Here are some examples:   sas-viya compute contexts list sas-viya batch contexts list   I am not aware of a sas-viya CLI method for modifying Connect contexts in SAS Viya Stable 2022.09.   Library Connections are created in SAS Studio, using the normal New Library button on the Libraries tab. Check the 'Assign and connect to data sources at startup' checkbox in the New Library dialog, and also the 'Allow all users to view the library connection' checkbox, to define the library at the Compute Context level, rather than just for yourself. The Library Connection is added to the list of resources associated with the compute context. Access to it is managed either in SAS Studio or in SAS Environment Manager > Contexts > Compute Contexts > [choose compute context] > Resources tab. See this post for details.   What is the scope of these changes?   Changes to a compute, connect or batch context affect all SAS Programming Run-time sessions started in that context, and don't affect sessions started in any other context.       User-specific   So far as I am aware, the only SAS client to explicitly support user-specific configuration for the SAS Programming Run-time is SAS Studio.   What can you change?   You can add user-specific SAS code statements to your own SAS Studio-specific autoexec file, which are run when you open SAS Studio and start a new compute session.   You can use the Custom SAS Code dialog to specify preamble and postamble SAS code statements to be run immediately before and after you run SAS Programs, and also optionally before and after running Tasks, Queries and Imports.   As hinted at earlier, when creating a Library Connections in SAS Studio, if you do not select the 'Allow all users to view the library connection', the library connection is referenced in your user-specific autoexec file and is thus available for you (only) but in ALL compute contexts. See  @GerryNelson's  post Managing Connections to Data Sources for Compute sessions in SAS Viya .   Where can you change it?   In SAS Studio, once a compute session has started, choose Options > Autoexec file from the application menu bar to show the Autoexec file popup dialog, to which you can add your own SAS code statements to run when your SAS Studio session starts, for example:     The content of your Autoexec file is loaded and runs automatically, once, when you (and only you) start a new SAS Studio compute session, in any compute context. It is not loaded or run for any other users.   In SAS Studio, once a compute session has started, you can also choose Options > Custom code to define your own preamble and/or postamble, and which sorts of SAS Studio tasks it should be run before and after. This custom preamble code runs before your submitted code, and the postamble code runs afterwards, each time you submit code in a SAS Studio session. Again, this applies to code run by you (and only you) in any compute context.   Here is some example preamble code:     And here is some example postamble code:     New Library Connections (also known as 'Connections to Data Sources' or 'Data Source Definitions') are created in the 'New Library' dialog in SAS Studio: open if by choosing the Libraries tab on the left, and then clicking on the 'New Library' button above the list of currently-defined libraries:     Check the 'Assign and connect to data sources at startup' checkbox in the New Library dialog. If you see it (because you are a SAS Administrator) do not check the 'Allow all users to view the library connection' checkbox. The library connection will be defined for only you. You will see its reference in your personal Autoexec file.   What is the scope of these changes?   These configuration settings only appear to affect the user who makes them, in SAS Studio (and not in other SAS clients or e.g. SAS batch programs) but for all Compute contexts.       If you know of other places to configure SAS Programming Run-time sessions, please comment below, or get in touch, and I'll update this post to mention them.   Many thanks to @RobCollum, @GerryNelson and @EdoardoRiva for their insightful comments and suggestions. I'd encourage you to read their posts, linked to from this one.   See you next time!   ... View more

In my last post, we looked at Managing monitoring data retention period and size in SAS Viya Monitoring for Kubernetes. In this post, let's find out how much log data is currently being stored by OpenSearch or ElasticSearch, how big the PVCs used to store it are individually and all together, and how to change the log data retention period.   In SAS Viya Monitoring for Kubernetes (v4m for short), log messages and their contextual data are gathered by Fluent Bit, streamed to OpenSearch and displayed in OpenSearch Dashboards. Versions of v4m earlier than version 1.2.0 used ElasticSearch and Kibana, which were essentially the same things. As we will see, both the new and old versions of these tools manage log data retention and size in the same way, so we can explain them together.     Incidentally, log retention is covered by recently updated documentation in SAS Help Center here, in place of the README files that were part of the SAS Viya Monitoring for Kubernetes project before. I think this is a big improvement.   What exactly does 3 days' log retention mean?   Contrary to what you might expect, old log data is not deleted as soon as it becomes more than 3 days old. Rather, whole days' of data are deleted at a time, once per day. Here's how it works in detail.   SAS Viya Monitoring for Kubernetes configures OpenSearch to keep log data in many separate indices, a bit like table partitions in other databases. There are separate indices for:   each UTC day each Kubernetes namespace (including separately for each Viya namespace if you have more than one) each application-level tenant (if you have more than one)   The 'one set of indices per UTC day' bit is most relevant for this post.   Each day at 00:00 UTC, OpenSearch stops writing log data to the current set of indices, and starts writing it to a new set. Separately, a job runs every few minutes in OpenSearch, and checks if the creation date of each index is older than 3 days ago. If the creation date of an index is older than that, the whole index is deleted. OpenSearch and ElasticSearch don't delete individual log messages from indices as they become older than 3 days, they delete an entire day worth of indexes, and thus that day's entire set of log messages, more or less all at once.   Example   With the default log data retention period of 3 days, let's imagine you look at the system at any time on 4th January UTC. OpenSearch will be writing log data as it arrives into the set of indices for 4th January. (There is a date in the index name).   The sets of indices which were created moments after midnight on 3rd and 2nd January, each containing all the log data from one of those dates, will still be stored in the PVC and loaded into memory.   However, just after midnight on 4th January, the set of indices containing all the log data collected on 1st January was created just over 3 days ago. So the set of indices for 1st January were deleted by a maintenance job, just after midnight UTC on 4th January. This means that in practice, the 3 day retention period results in log data being available for the past 2 days plus the time since the most recent midnight UTC.   One more thing: in contrast to Prometheus's metric data, there is no enforced limit to log data size, other than the size of the PVC used to store the log data. It is quite possible to fill that PVC, which causes OpenSearch to stop working properly, so we need to know how to avoid that happening.     See how much log data storage is currently being used   The method for seeing how much storage is available and how much is actually being used depends on the type of Storage Class used for your Persistent Volume Claims. In our workshop environments we use a simple NFS share, but in a production environment you should use something better, like Azure Files or S3.   Note: NFS storage is not ideal in a cloud environment, and we don't recommend it for production deployments. However, it is easy (and free) to use in our RACE-based classroom and lab environments, and it is therefore the only persistent Storage Class currently available in them.   In our workshop environments, the default (and only) storage class is nfs-client . The files written to PVCs are ultimately written to a directory structure in the filesystem on the sasnode01 host in each collection, since that is where the cluster's NFS server runs. From a shell on the sasnode01 host, we can browse that filesystem and find the data under /srv/nfs/kubedata . This is highly implementation-specific. There is little chance a customer deployment would be set up like this. Talk to your architect or Kubernetes administrator, and they may be able to suggest something similar to the following that makes sense in your environment.     How big are the log data PVCs?   Find the specified number and size of PVCs   By default, our SAS Viya Monitoring for Kubernetes project configures and deploys OpenSearch or ElasticSearch with three pods in its v4m-search statefulset (OpenSearch, v4m 1.2.0 later) or its v4m-es-data statefulset (ElasticSearch, v4m 1.1.8 and earlier).   Each of these three pods has its own PVC. The aggregate storage size of these three PVCs is the total storage available to OpenSearch or ElasticSearch. Exactly what data is stored in these PVCs varies slightly between the two versions - OpenSearch uses just one set of PVCs for all of its data, whereas ElasticSearch has a second, smaller set of PVCs for 'master data', which I presume stores provided and user-created objects and other management data, but our focus here will be on the main data PVCs.   Our SAS Viya Monitoring for Kubernetes project is configured to request each of those PVCs to be 30 GiB (30 Gibibytes = 30 x 2 30 bytes). So across the 3 pods, by default OpenSearch or ElasticSearch requests 3 x 30 GiB = 90 GiB storage in total.   You can see how this is specified either in the user-values-opensearch.yaml file (OpenSearch) or the user-values-elasticsearch-open.yaml file (ElasticSearch) in the v4m USER_DIR/logging directory, or in the default helm charts referenced at the top of each of those files, if those files don't override the defaults. Look for a value for replicas for how many pods there should be in the statefulset, and (often rather separately) a persistence section containing a size value which may be something like 30Gi . So, in theory, that's what we should actually have.     Find the actual number and size of PVCs   In theory, there's no difference between theory and practice. But in practice, there is. You can find the actual size of the PVCs in the monitoring namespace in practice using kubectl or Lens. In kubectl, try something like this (where logging is the namespace where the log monitoring components of SAS Viya Monitoring for Kubernetes are deployed): kubectl get pvc -n logging   Look for the number and size of the v4m-search-v4m-search-* PVCs (OpenSearch) or the data-v4m-es-data-* PVCs (ElasticSearch).   In one of our workshop deployments with v4m 1.2.1, the logging (or if you prefer, log monitoring) namespace is v4mlog , so here's what that looks like for OpenSearch:   [cloud-user@hostname logging]$ kubectl get pvc -n v4mlog NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE v4m-search-v4m-search-0 Bound pvc-82514968-1728-4bc8-95b7-708b08043af1 30Gi RWO nfs-client 25h v4m-search-v4m-search-1 Bound pvc-82ffb122-1730-4780-825c-ce8887e0b01f 30Gi RWO nfs-client 25h v4m-search-v4m-search-2 Bound pvc-47e35a8e-423c-4490-bbb7-a17e5e0def17 30Gi RWO nfs-client 25h   In another of our workshop deployments with v4m 1.1.8, this is the equivalent for ElasticSearch:   [cloud-user@hostname ~]$ kubectl get pvc -n v4mlog NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-v4m-es-data-0 Bound pvc-ed9314dc-06b4-471a-9a37-499cf42ddcc9 30Gi RWO nfs-client 19d data-v4m-es-data-1 Bound pvc-4127753a-528b-40ec-93c8-3826c6397d5c 30Gi RWO nfs-client 19d data-v4m-es-data-2 Bound pvc-804a632c-97a4-4e7b-942d-55ba6fd6828c 30Gi RWO nfs-client 19d data-v4m-es-master-0 Bound pvc-18252229-6e88-4e44-9037-15ccd68664f0 8Gi RWO nfs-client 19d data-v4m-es-master-1 Bound pvc-2047fd01-0c15-4f90-842a-a86129647580 8Gi RWO nfs-client 19d data-v4m-es-master-2 Bound pvc-a68baa6b-fde8-47d4-9de7-2970d0be9d48 8Gi RWO nfs-client 19d   From this we can see that the size of the total data storage for OpenSearch or ElasticSearch is 3 x 30Gi = 90Gi.   Here is the OpenSearch PVC set in Lens:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Either of these ways to see the PVC size should work on all environments, if you have kubectl or Lens, a kube config file which lets you access the cluster, and you know your log monitoring namespace name.     How much data is being stored in the log data PVCs?   So we know how big the PVCs are. How much of that space is being used?   This is more implementation-specific. Since we are using NFS for our PVCs, as explained above the data ends up on sasnode01 under /srv/nfs/kubedata . Let's look at how to find the amount of data stored on our NFS share - the actual values below are not typical; they are likely much smaller than you should expect in a real production environment.     PVC use by OpenSearch in v4m 1.2.0 and later   If you have v4m version 1.2.0 or later, something along the lines of this command might show how big the data in the three OpenSearch PVCs actually is:   for f in /srv/nfs/kubedata/v4mlog-v4m-search-v4m-search-* ; do sudo du -xsh $f; done   Substitute the path to the kubedata directory in your NFS shared volume in place of /srv/nfs above. Here is some example output from a very lightly-used workshop environment:   [cloud-user@hostname logging]$ for f in /srv/nfs/kubedata/v4mlog-v4m-search-v4m-search-* ; do sudo du -xsh $f; done 4.5G /srv/nfs/kubedata/v4mlog-v4m-search-v4m-search-0-pvc-82514968-1728-4bc8-95b7-708b08043af1 3.1G /srv/nfs/kubedata/v4mlog-v4m-search-v4m-search-1-pvc-82ffb122-1730-4780-825c-ce8887e0b01f 5.7G /srv/nfs/kubedata/v4mlog-v4m-search-v4m-search-2-pvc-47e35a8e-423c-4490-bbb7-a17e5e0def17   So in this environment, there is currently 4.5G + 3.1G + 5.7G = 13.3GiB of log data. Well, actually, slightly less than that. These PVC directories also contain a few MB of other stored data. Remember this example is intended to show how to calculate disk usage on our NFS storage, not to give estimates of typical usage.     PVC use by ElasticSearch in v4m 1.1.8 and earlier   If you have v4m version 1.1.8 or earlier, something along the lines of this command might show how big the data in the three ElasticSearch data PVCs actually is:   for f in /srv/nfs/kubedata/v4mlog-data-v4m-es-data-* ; do sudo du -xsh $f; done   Substitute the path to the kubedata directory in your NFS shared volume in place of /srv/nfs above. Here is some example output from a very lightly-used workshop environment:   [cloud-user@hostname ~]$ for f in /srv/nfs/kubedata/v4mlog-data-v4m-es-data-* ; do sudo du -xsh $f; done 1.3G /srv/nfs/kubedata/v4mlog-data-v4m-es-data-0-pvc-ed9314dc-06b4-471a-9a37-499cf42ddcc9 511M /srv/nfs/kubedata/v4mlog-data-v4m-es-data-1-pvc-4127753a-528b-40ec-93c8-3826c6397d5c 854M /srv/nfs/kubedata/v4mlog-data-v4m-es-data-2-pvc-804a632c-97a4-4e7b-942d-55ba6fd6828c   So in this environment, there is currently 1.3G + (511/1024)G + (854/1024)G = 2.63GiB of log data. Again, this is meant to show the method for calculating usage, not to give estimates of typical usage.     Alternative - look at the OpenSearch Dashboards > Index Management > Indices page   Another way to roughly estimate log data size - and one which does not require command line access to the servers or a kube config file - is to open the OpenSearch Dashboards (or Kibana) Index Management page, and switch to the Indices tab. This tab shows a table of all Indices currently held in OpenSearch (or ElasticSearch), with statistics for each index including its total size:     With a bit of patience, you could manually add up the 'Total size' values of the each row. The table is paged, so make sure you include each page. It is possible that there is an API or command line way to do this - I have not explored OpenSearch or ElasticSearch APIs or command-line tools.     How fast will it grow?   The rate at which log data is collected can be quite volatile over time, as it depends very much on how heavily SAS Viya and other applications in the Kubernetes cluster are used, as well as on whether log thresholds are changed to e.g. increase log detail and for how long. Extrapolating log data growth over time is therefore more of an art than a science, but it's worth estimating (or guess-timating) based on what you know about how your collection's level of activity now, historical log data size and your understanding of how you are likely to change logging levels. Then monitor log data size closely. This may reveal that your early estimates are quite inaccurate, but it's better than not estimating and monitoring it at all.   See the Current Log Data Retention Period   To see the current log data retention period, follow these steps:   Find the URL for OpenSearch Dashboards or Kibana in your deployment of SAS Viya Monitoring for Kubernetes. These tips may help: In our workshop environments: If v4m version 1.2.0 or later is deployed, the OpenSearch Dashboards URL is https://osd.ingress_controller_hostname/ where ingress_controller_hostname  is the full hostname of the sasnode01 host in our Kubernetes cluster. For example, in one RACE collection I happen to have running, this is http://osd.pdcesx02020.race.sas.com . We are considering changing the hostname prefix from osd to something else to avoid confusion with SAS/ODS, so if you don't find it this way, check the workshop instructions for where to find OpenSearch Dashboards. If v4m 1.1.8 or earlier is deployed, the Kibana URL is  https://kibana.ingress_controller_hostname/ where ingress_controller_hostname  is the full hostname of the sasnode01 host in our Kubernetes cluster. A more typical default URL for OpenSearch Dashboards is  something like http://ingress_controller_hostname/dashboards  where ingress_controller_hostname  is the hostname of your Kubernetes cluster's ingress controller. A more typical default URL for Kibana is  something like http://ingress_controller_hostname/kibana  where ingress_controller_hostname  is the hostname of your Kubernetes cluster's ingress controller. If you are not sure, try running kubectl get ingress -n logging where logging is the namespace in which the SAS Viya Monitoring for Kubernetes logging (or log monitoring) components are deployed. Look for an ingress named v4m-osd  or v4m-es-kibana-ing depending on whether you have v4m 1.2.0 or later, or 1.1.8 or earlier. You can also find this in Lens, under Network > Ingresses, and filter to your monitoring namespace. Again, look for an ingress named  v4m-osd  or v4m-es-kibana-ing . In your web browser, browser to OpenSearch Dashboards or Kibana, and sign in as admin . If you are prompted to select your tenant, choose the most appropriate tenant (there may be no choice other than cluster_admins ), check the checkbox for 'Remember my selection next time I log in from this device' if there is one, and click Confirm. In OpenSearch Dashboards or Kibana, use the navigation menu on the left to browse to the Index Management page. You may have to scroll down to see it. On the Index Management page, the default tab is the Index policies tab. On that tab, click viya_logs_idxmgmt_policy:     Look at the policy. In OpenSearch Dashboards, you need to scroll down to the States section, and expand the 'hot' state to see when indices in this state are meant to transition to the 'doomed' state. It shows that this is to happen when the 'Minimum index age is 3d':   For ElasticSearch, you just see a JSON representation of the policy, but it is quite readable:   Here, ' 3d ' means 3 days. The only units that are sensible in the context of our SAS Viya Monitoring for Kubernetes configuration are whole days.   Change Log Data Retention Period   There are two steps to changing the log data retention period. First, change the index management policy, and optionally second, re-apply that policy to existing indices. Let's start by changing the policy. From the page where you see the index management policy details, follow these steps: Click the Edit button to edit the index management policy. The user interfaces for editing the index policies differ between OpenSearch and ElasticSearch: OpenSearch/OpenSearch Dashboards offers you a choice of a visual editor, or JSON editor. The visual editor is okay, if slightly convoluted to use. The JSON editor is less slick, but I find it quicker and more straightforward to use. In the index management policy detail page, click the Edit button top right, and choose the JSON editor. ElasticSearch/Kibana only has a JSON editor. Clicking the Edit button opens it. The 'Edit policy' page is similar for both OpenSearch and ElasticSearch. Scroll down to the JSON document, find "min_index_age": "3d" , and change it to a new value, e.g. "min_index_age": "4d" . Then click Update. Read the OpenSearch or ElasticSearch documentation if you want to know more about making your own index management policy; that is beyond the scope of this post. We are only trying to change the retention period in an existing policy. Next, apply the changed policy to existing SAS Viya indices: In either OpenSearch Dashboards, or ElasticSearch/Kibana, switch to the Managed Indices tab in the Index Management page. Click the 'Change policy' button in the top right corner of this page. In the Managed indices filter box, type viya_logs- , and accept the suggested value of viya_logs-* .   Below this (scroll down if necessary), under Choose new policy, select viya_log_idxmgmt_policy from the dropdown list.   Click the Change button, on the right. A popup message should appear saying that some number of indexes were changed. When you change the log retention period interactively like this, it is recommend to also update the LOG_RETENTION_PERIOD value in your in USER_DIR/logging/user.env to keep it consistent with this change. This is sensible, because if you later remove and redeploy the v4m logging stack, your new configuration (which in this case keeps indices for e.g. 4 days) is preserved, instead of reverting to the original configuration (which keeps indices for 3 days). We made a similar recommendation for metric data in my previous post.   Hopefully this gives a reasonably complete explanation of finding the PVC size and usage, and finding and changing the log data retention period. I have intentionally not covered changing the PVC size, in either this post or my previous one. That is a sufficiently complex topic to explain that it deserves a post of its own, mostly because the type of storage class used for your PVCs greatly affects the method for changing its size, and also greatly effects whether you can change the PVC size without dropping and re-creating all the data currently stored in it. See you next time!   Find more articles from SAS Global Enablement and Learning here. ... View more

In SAS Viya Monitoring for Kubernetes, metric data is gathered by Prometheus and displayed in Grafana.  As this blog post explains, Prometheus will keep metric data until either the amount of data it has exceeds a specified retention size, at which point it will begin deleting the oldest data to make way for new metric data, or until that metric data exceeds the specified retention time. If the retention time limit is reached first, the metric data older than that limit will be deleted even if the total size of data retained is smaller than the retention size.   In this post, we'll see how to calculate how much space is being used to store Prometheus metric data in your cluster, and how to change both the retention period and retention size.   Incidentally, my colleague  Raphaël Poumarede (@RPoumarede) has written here recently about managing many aspects of storage, in his posts Take care of your Viya storage before it takes care of you – Part 1: Planning ahead and anticipating and Take care of your Viya storage before it takes care of you – Part 2: purge and expand. They are fantastic posts - do read them!   See how much metric data storage is currently being used (NFS)   The method for seeing how much storage is available and how much is actually being used depends on the type of Storage Class used for your Kubernetes Persistent Volume Claims (PVCs). In our internal GEL workshop environments we use a simple NFS share, but in a production environment you should use something better, like Azure Files or S3.   Note: NFS storage is not ideal in a cloud environment, and we don't recommend it for production deployments. However, it is easy (and free) to use in our RACE-based classroom and lab environments, and it is therefore the only persistent Storage Class currently available in them. In our workshop environments, the default (and only available) storage class is nfs-client . The files written to PVCs are ultimately written to a directory structure in the filesystem on the sasnode01 host in each collection, since that is where the cluster's NFS server runs. From a shell on the sasnode01 host, we can browse that filesystem and find the data under /srv/nfs/kubedata . This is highly implementation-specific. There is little chance a customer deployment would be set up like this. Talk to your architect or Kubernetes administrator, and they may be able to suggest something similar to the following that makes sense in your environment.     How big is the metric data PVC?   You can find the size of the PVCs in the monitoring namespace using kubectl or Lens. In kubectl, try something like this (where monitoring is the namespace where the monitoring components of SAS Viya Monitoring for Kubernetes are deployed): kubectl get pvc -n monitoring   And look for the size of the PVC for Prometheus. In our workshop deployments, the monitoring namespace is v4mmon , so here's what that looks like - scroll to the right to see the PVC capacities, i.e. their size:   [cloud-user@rext03-0272 ~]$ kubectl get pvc -n v4mmon NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE alertmanager-v4m-alertmanager-db-alertmanager-v4m-alertmanager-0 Bound pvc-5dfcb9bf-984d-4d19-911d-a573cd6390b0 10Gi RWO nfs-client 17h prometheus-v4m-prometheus-db-prometheus-v4m-prometheus-0 Bound pvc-5538cdf0-a591-462c-bfb1-dc7d5b37b12c 25Gi RWO nfs-client 17h v4m-grafana Bound pvc-512d42ac-ae98-4813-a8d0-c377f0fb3738 5Gi RWO nfs-client 17h   From this we can see that the size of the Prometheus PVC is 25Gi. Here is the same thing in Lens:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.</p   Either of these ways to see the PVC size should work on all environments, if you have kubectl or Lens, a kube config file which lets you access the cluster, and you know your monitoring namespace name.     How much data is being stored in the metric data PVC?   So we know how big the PVC is, but how much of it is being used, and when the system has been running as long as the metric data retention period (which we will come to in a moment), how much of that space is likely to be used?   This is more implementation-specific. Since we are using NFS for our PVCs, and as explained above the data ends up on sasnode01 under /srv/nfs/kubedata , this command will show how big the data in the Prometheus PVC actually is:   for f in /srv/nfs/kubedata/v4mmon-prometheus* ; do sudo du -xsh $f; done   Substitute the path to the kubedata directory in your NFS shared volume in place of /srv/nfs above. Here is some example output from a very lightly-used workshop environment:   [cloud-user@rext03-0272 monitoring]$ for f in /srv/nfs/kubedata/v4mmon-prometheus* ; do sudo du -xsh $f; done 1.9G /srv/nfs/kubedata/v4mmon-prometheus-v4m-prometheus-db-prometheus-v4m-prometheus-0-pvc-5538cdf0-a591-462c-bfb1-dc7d5b37b12c   So in this environment, there is currently 1.9Gi of monitoring data, 18 or so hours after the environment started up, in a PVC with a nominal capacity of 25Gi. So it's currently something like 8% used.   The rate at which metric data is collected is fairly constant over time, so we can extrapolate that after 7 days, we would have ((2 / 17) * 24 * 7) = approx 20 Gi data, and the PVC might reach about 80% full.   However, as Raphaël explains in his post, when you use an NFS server as the backend of your PersistentVolumeClaims (PVCs): the claimed size is not enforced as a limit. So if the stored metric data were to grow over 25Gi in size, Kubernetes would not do anything to prevent it growing to fill the disk. This is one of several reasons why it is not optimal to use a single shared NFS mount for all our PVCs, or really to use NFS at all!   Fortunately, Prometheus has a feature called retention size that will limit the size of the metric data kept to a maximum of 20GiB by default. This limit can easily be changed. This means that even when it's not practical to change the Prometheus PVC size to manage the storage space used by Prometheus, you can still control how much space it uses quite effectively.     See the Current Metric Data Retention Period and Retention Size   To see the current metric data retention period, follow these steps:   Find the URL for Prometheus in your deployment of SAS Viya Monitoring for Kubernetes. These tips may help: In our internal GEL workshop environments, the Prometheus URL is https://prometheus.ingress_controller_hostname/ where ingress_controller_hostname  is the full hostname of the sasnode01 host in our Kubernetes cluster. A more typical default URL for Prometheus is  something like http://ingress_controller_hostname/prometheus where ingress_controller_hostname  is the hostname of your Kubernetes cluster's ingress controller. If you are not sure, try running kubectl get ingress -n monitoring where monitoring is the namespace in which the SAS Viya Monitoring for Kubernetes monitoring components are deployed. Look for an ingress named v4m-prometheus . You can also find this in Lens, under Network > Ingresses, and filter to your monitoring namespace. Again, look for an ingress named v4m-prometheus . In your web browser, open the Prometheus Command-Line Flags page, at Prometheus_URL/flags where Prometheus_URL is the URL you found in the previous step. In our workshop environments, this is https://prometheus.ingress_controller_hostname/flags n the Prometheus Command-Line Flags page, in the 'Filter by flag, name or value' textbox, enter the string:  --storage.tsdb.retention   The value of --storage.tsdb.retention.time in the screenshot above is 1w meaning 1 week, and the value of --storage.tsdb.retention.size is 20GiB. Those are the defaults, but they may have other values in your deployment.   Aside: the flag --storage.tsdb.retention has been deprecated since Prometheus 2.8; the very earliest release of SAS Viya Monitoring for Kubernetes used Prometheus 2.21.0 (or thereabouts - in ops4viya version 0.1.0), so this flag has been deprecated since long before SAS Viya Monitoring for Kubernetes was first released.   Change Metric Data Retention Period and Retention Size: quick but non-persistent method   As Raphaël explains in his blog post, (and this is very similar to a process described in the Logging stack's Log_Retention.md) you can quickly change the Prometheus storage retention time and size by running Kubectl patch commands something like this, specifying your monitoring namespace in place of our v4mmon (scroll the sample code below to the right to see the new values in bold😞   kubectl -n v4mmon patch prometheus v4m-prometheus --type merge --patch '{ "spec": { "retention": "2d" }}' kubectl -n v4mmon patch prometheus v4m-prometheus --type merge --patch '{ "spec": { "retentionSize": "5GiB" }}'   Thanks to the Prometheus operator, this change is noticed and the pods are restarted with the new values. However, if you do not also change the retention period and retention size in your ${USER_DIR)/monitoring/user-values-prom-operator.yaml file as described in the next section, they would revert to the values defined in that file if you ever undeploy and redeploy the monitoring stack, for example when you scale your SAS Viya deployment down to save resources when you are not using it.   To change to the values in a way which will persist across an undeployment and redeployment, follow the process below as well as (or instead of) patching Prometheus's pod spec.     Change Metric Data Retention Period and Retention Size: slower but persistent method   To change the metric data retention period and/or retention size (note: not the PVC storage size) so that the changes would persist if you undeploy and redeploy the monitoring stack, follow these steps: If you have not installed SAS Viya Monitoring for Kubernetes yet, follow the installation steps as far customizing the deployment of the monitoring components, but return here before deploying them. Some notes about this: An official set of instructions are here: https://github.com/sassoftware/viya4-monitoring-kubernetes/blob/master/monitoring/README.md. Documentation for the customization and deployment process for the monitoring components can also be found in the SAS Viya Administration Guide. Follow the steps under 'Perform Pre-Deployment Tasks', including 'Create a Local Copy of the Repository' and 'Customize the Deployment' in the project README.md, or up the point in the Administration Guide where you are ready to run monitoring/bin/deploy_monitoring_cluster.sh . You may have a different method for deploying the project - that's fine. Once you have performed the pre-deployment tasks, you should know the location of your viya4-monitoring-kubernetes directory, and the location of your local customization files, otherwise known as your USER_DIR . Alternatively, if the monitoring components of SAS Viya Monitoring for Kubernetes are already deployed, find the path of the viya4-monitoring-kubernetes directory, and the USER_DIR directory on the host machine from which you are deploying SAS Viya Monitoring for Kubernetes.  These tips may help: Identify the host machine used to deploy SAS Viya Monitoring for Kubernetes. In our workshop environments, this is sasnode01 . In our workshop environments, the viya4-monitoring-kubernetes directory is /home/cloud-user/viya4-monitoring-kubernetes , and the USER_DIR directory is /home/cloud-user/.v4m on sasnode01 . Ask to whoever deployed SAS Viya Monitoring for Kubernetes where these two directories are - they should know. If you know the command or script that was or will be used to deploy SAS Viya Monitoring for Kubernetes, look for the path preceding the deployment scripts, and also look for something defining an environment variable USER_DIR and giving it a value of a filesystem path. Try running a command like locate user.env on the machine you are using to deploy SAS Viya Monitoring for Kubernetes.  Of the many directories this may reveal, one might be your USER_DIR , and you might recognize it when you see it. In your USER_DIR directory, look for a file at ${USER_DIR)/monitoring/user-values-prom-operator.yaml . If you don't see one, copy viya4-monitoring-kubernetes/monitoring/user-values-prom-operator.yaml  to  ${USER_DIR)/monitoring/user-values-prom-operator.yaml . Edit your ${USER_DIR)/monitoring/user-values-prom-operator.yaml file, find an uncommented section that looks roughly like this - after previous customizations the values and perhaps even the structure will differ from this: prometheus: enabled: true prometheusSpec: externalUrl: http://host.mycluster.example.com:31090 retention: 7d retentionSize: 20GiB storageSpec: volumeClaimTemplate: spec: storageClassName: myCustomStorageClass resources: requests: storage: 25Gi If you don't have an uncommented section like that in your ${USER_DIR)/monitoring/user-values-prom-operator.yaml file, copy the existing commented-out section, then uncomment and keep only the lines in bold above, and their 'parent' lines, e.g.: prometheus: prometheusSpec: retention: 7d retentionSize: 20GiB Note that correct indentation in YAML is important. Edit the values for retention , and if you wish, retentionSize . Note especially that the units for retentionSize are indicated with a suffix of GiB (Gigbibytes, meaning 2^30 bytes), but the units for storage are indicated with a suffix of Gi (which also means Gibibytes!). Thanks to my colleague Rob Collum for setting me straight on this point: the two suffixes here (GiB and Gi) are both power-of-two indicators, both have the same meaning: both indicate Gibibytes. If Gigabytes had been meant (and it isn't), the unit might be indicated as GB (and it is not). The point is, pay attention to the unit suffixes - they differ, but mean the same thing. Note that I am not advising you to change the storageSize. Resizing an existing PVC without deleting and re-creating it (and thereby losing all the data in it) is not supported for all Kubernetes Storage Classes. Even if you are using a storage class for which AllowVolumeExpansion = true , you also need a storage provisioner plug-in in your cluster that can resize the volumes. In our workshop environment, with the nfs-client storage class, just specifying a new volume size like this did not work: kubectl -n v4mmon patch pvc prometheus-v4m-prometheus-db-prometheus-v4m-prometheus-0 -p '{"spec":{"resources":{"requests":{"storage":"35Gi"}}}}' In our workshop environment, the nfs-client storage class has AllowVolumeExpansion = true , and when I try to expand the Prometheus PVC by specifying a larger size, Lens reports the new larger size. But in fact the PVC does not actually change size and we get a Kubernetes error event for the PVC saying " Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC. ". Therefore, at present the only method I have been able to successfully use for changing the size of the Prometheus PVC is to completely uninstall the entire monitoring stack, delete ALL of the monitoring namespace PVCs (in fact I prefer to just delete the whole namespace because it is simpler), and re-deploy the monitoring stack again with the new Prometheus storage size. Obviously, deleting the PVCs destroys any previously collected metric data, and any other user generated content such as custom Grafana dashboards etc. , so I do not recommend you do it unless you are happy to lose that data. If you are confident that you know how to resize PVCs using the storage class used for the Prometheus PVC in your cluster, then this process can be non-destructive of data and user content. But I have not been able to test it successfully in my environment, and therefore am not describing the steps here. You may also need to modify other lines in both this section of this file, and in other areas of the file, for example to define the storage class, or ingress. See the pre-deployment customization instructions. Save your changes to ${USER_DIR)/monitoring/user-values-prom-operator.yaml . Now you can follow your usual deployment steps to re-deploy (or deploy) the cluster-level monitoring components, e.g.: cd ~/viya4-monitoring-kubernetes/ export USER_DIR=/path/to/your/user_dir/directory # Used by v4m scripts ./monitoring/bin/deploy_monitoring_cluster.sh In our workshop environments, redeploying the cluster level monitoring components of SAS Viya Monitoring for Kubernetes typically takes 1 to 2 minutes.   Assess whether your changes took effect   Follow the steps under 'See the Current Metric Data Retention Period and Retention Size' above to see if the retention period and retention size changed as you intended. Hopefully, they did! If not, review the steps above and make sure you followed them correctly.   By following the steps in this post, you should be able to fully manage the storage space used by your SAS Viya Monitoring for Kubernetes instance of Prometheus, and have it retain metric data for the optimum length of time that your available storage permits.     Many thanks to my colleague Raphaël Poumarede ( @RPoumarede ) for his help with parts of this post, and to my colleague Rob Collum ( @RobCollum ) for correcting an error in an earlier version of the note on GiB vs Gi (which are both Gibibytes, 2^30 bytes or 1024^3 bytes) vs GB (which is Gigabytes, where 1GB = 1000 MB = 1000^3 bytes = 10^9 bytes).   See you next time!   ... View more

On 14 June 2022, the team behind the SAS Viya Monitoring for Kubernetes project (let's call it v4m for short) released version 1.2.0. It's a major release with several changes over v4m 1.1.8, described on the version 1.2.0 release page. Two changes I want to highlight are: v4m dropped support for Kubernetes 1.20. To update v4m to version 1.2.0 you need to be on Kubernetes 1.21 or higher (and of course meet the other dependencies for the version of SAS Viya you are deploying). If you are on Kubernetes 1.20, you can still deploy v4m version 1.1.8. The logging stack in v4m 1.2.0 now uses OpenSearch 1.3.2 instead of Open Distro for Elasticsearch (ODfE).   You can read a nice explanation here, but basically ODfE is no longer releasing new versions, OpenSearch is a fork of ElasticSearch which is being actively developed and supported by Amazon, so obviously that's the one the v4m project should be using. And now it is. To an end user the two look and behave similarly and the user experience transition is smooth. The development team have documented Differences between Open Distro for Elasticsearch and OpenSearch in great detail, if it's important for you to know. The team have also gone to considerable effort to make the upgrade process smooth and simple.   However, if it's your job to deploy v4m, I'd like to share a couple of things I learned while updating the code we maintain to deploy SAS Viya Monitoring for Kubernetes (or v4m) in our GEL workshop environments. Perhaps they will save someone a bit of head-scratching or a call to support.   For context, there are several ways to deploy SAS Viya Monitoring for Kubernetes (v4m), and I really only ever deploy it in one of those ways - I maintain a script which, when run in a new set of host machines will clone the v4m project from GitHub, create the customization files we decided to use containing our preferred settings, and call the v4m project's monitoring and logging deployment scripts. If you deploy SAS Viya Monitoring for Kubernetes as part of using the SAS Viya 4 Deployment project, I expect all this should be taken care of for you. If you deploy it into Red Hat OpenShift, I'm not sure how much the changes below apply to you, as I have barely any experience of OpenShift.   Bear in mind the v4m project has two main solutions (or modules, or stacks - the terminology varies a bit); monitoring and logging. This post concentrates on the logging solution.     Logging customization file name changes and a new file for OpenSearch Dashboards   The table below lists list the logging solution customization files our script creates just before it deploys v4m versions 1.1.8 and 1.2.0. In this table ${USER_DIR} means a shell variable containing the location of our v4m customization files.   v4m 1.1.8 v4m 1.2.0 Description ${USER_DIR}/logging/user.env ${USER_DIR}/logging/user.env We use this to specify the logging namespace name, log retention period, the admin password, the logadm password, the nginx namespace and service name. There are no important changes to this file at this release.   ${USER_DIR}/logging/user-values-osd.yaml New for v4m 1.2.0, we use this to specify the ingress configuration for OpenSearch Dashboards. ${USER_DIR}/logging/user-values-elasticsearch-open.yaml ${USER_DIR}/logging/user-values-opensearch.yaml Note: this configuration file's name changed for version 1.2.0. We use this to specify ElasticSearch or OpenSearch's log data PVC size, and ingress configuration. For v4m 1.1.8, this file covered the ingresses for both ElasticSearch and Kibana. For v4m 1.2.0 this file only covers the ingress for OpenSearch. ${USER_DIR}/logging/user-values-es-exporter.yaml tee ${USER_DIR}/logging/user-values-es-exporter.yaml We use this only to specify an image pull policy. Note: despite switching from ElasticSearch to OpenSearch, the two-letter abbreviation in this filename remains 'es'.   There are several ways in which you might customize your deployment of the logging solution in v4m and the table above just represents how we needed and wanted to customize it for our GEL workshop student hands-on environments in RACE.     HTTPS or HTTP for OpenSearch and OpenSearch Dashboards   SAS Viya Monitoring for Kubernetes uses Helm for deployment.   The yaml files used to configure the logging and monitoring solutions in SAS Viya Monitoring for Kubernetes contain values that override defaults (or provide values where there is no default) in those Helm charts. Therefore we sometimes see the absence of a value in a configuration file as meaning 'use the default'. It's a somewhat complex topic that candidly, I'm not an expert on.   However, I learned that whether or not we are using Transport Layer Security (TLS, using the HTTPS protocol) or not (using the HTTP protocol), HTTPS must be specified for the ElasticSearch and OpenSearch components of the logging solution. It will not work otherwise.   Here's how that worked out for our setup.   Version 1.1.8   In the deployment configuration files we created for v4m 1.1.8, ${USER_DIR}/logging/user-values-elasticsearch-open.yaml covered the ingresses for both ElasticSearch and Kibana. As part of this, we specified that HTTPS should be used for ElasticSearch, and we did not specify either way whether HTTP or HTTPS should be specified for Kibana, which results in Kibana not using TLS, and using HTTP:   elasticsearch: imagePullPolicy: "IfNotPresent" data: persistence: size: ${_logDataSizePerNodeGiB}Gi client: ingress: annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/backend-protocol: HTTPS enabled: true hosts: - elasticsearch.$(hostname -f) kibana: imagePullPolicy: "IfNotPresent" service: type: ClusterIP nodePort: null ingress: annotations: kubernetes.io/ingress.class: nginx enabled: true hosts: - kibana.$(hostname -f)   Of course, you absolutely can and probably should use TLS for Kibana - but in our low security classroom deployments, we hadn't had any pressing need to do it.     Version 1.2.0   In the deployment configuration files we created for v4m 1.2.0, ${USER_DIR}/logging/user-values-osd.yaml covers the ingresses for OpenSearch Dashboards (equivalent to Kibana), and ${USER_DIR}/logging/user-values-opensearch.yaml now only covers OpenSearch (equivalent to ElasticSearch).   Note: Now that each of these files only configures one product, the 'top level' yaml keys ('elasticsearch' and 'kibana') in the earlier user-values-elasticsearch-open.yaml are no longer necessary, and the keys that used to be beneath them are all at the top level. user-values-opensearch.yaml was easy enough:   image: pullPolicy: "IfNotPresent" persistence: size: ${_logDataSizePerNodeGiB}Gi ingress: enabled: true annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/backend-protocol: HTTPS hosts: - opensearch.$(hostname -f)   However, having seen an HTTPS in the ElasticSearch configuration file for v4m 1.1.8, I initially created a user-values-osd.yaml which similarly specified HTTPS for OpenSearch Dashboards, despite having not otherwise configured TLS for our logging or monitoring stacks:   ingress: enabled: true annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/backend-protocol: HTTPS hosts: - host: osd.$(hostname -f) paths: - path: /   When I tried browsing to osd.the-relevant-ingress-controller-hostname.sas.com, I got an 502 Bad Gateway error.   The solution turned out to be simple but was not obvious to me at the time - though in hindsight it is a little more obvious - we haven't configured TLS (HTTPS) for our logging or monitoring stacks, so we should access them over HTTP, and our user-values-osd.yaml should tell the ingress to use HTTP, like this:   ingress: enabled: true annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/backend-protocol: HTTP hosts: - host: osd.$(hostname -f) paths: - path: /   Thanks as ever to my colleague Greg Smith for spotting my error.   The learning point is to configure your ingress to use always use HTTPS for ElasticSearch or OpenSearch, as they only work using TLS for internal interprocess communication, but to configure external access to your Kibana or OpenSearch dashboards with HTTPS or HTTP according to whether or not you have set up TLS-enabled Monitoring and Logging. And, when you haven't, be aware of it!     Deployment script name change The final change to the logging stack at 1.2.0 is that if you used to call logging/bin/deploy_logging_open.sh (where the 'open' referred to OpenDistro for ElasticSearch) from  within the viya4-monitoring-kubernetes project directory, you should now call logging/bin/deploy_logging.sh. Deployment on Red Hat OpenShift still uses its own separate deployment scripts, which I can't comment on.     No changes to deployment of the monitoring part of v4m at version 1.2.0   I did not need to make any modifications to the parts of our GEL workshop setup script that deploys the v4m monitoring solution, as we updated from deploying v4m 1.1.8 to v4m 1.2.0. That carried on working just the same as before.   I hope sharing this helps someone else avoid the simple mistakes I made, and thereby saves them a bit of time. Overall, the process was not complicated at all, and I like the new version and its clean new look.   See you next time!   Find more articles from SAS Global Enablement and Learning here. ... View more