Manage your SAS Viya platform configuration in Google Cloud Source Repositories
As a follow-up of my previous post "Manage your SAS Viya platform configuration in a Git Repository", I wondered if it would be possible to store and manage the SAS Viya configuration in a Cloud Provider’s Git repository service (instead of using something like GitLab or GitHub).
Git is the most widely used source-code management tool and Git repositories have become a key component of modern DevOps platforms. Azure, AWS and Google Cloud Platform offer "git compatible" Code Source repository services, such as Azure Repos, AWS CodeCommit or Google Cloud Source Repositories.
But can the SAS Viya platform play nicely with them ?
Here is a diagram of the generic deployment system that we’ll aim to put in place.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Assuming we use the "sas-orchestration command" deployment method and already have a Cloud Source Repository in place (created from the Cloud portal or with the CLI), our goal is :
- To push an initial SAS Viya configuration (a folder containing the kustomization.yaml file and the site-config folder) to the Cloud Git Repository.
- Then to run the sas-orchestration command to deploy SAS Viya using the configuration files stored in the Cloud Git Repository.
The purpose of this post is to demonstrate, in a few steps, how to do that with the Google Cloud Source Repositories service.
Introducing the Google Cloud Source Repositories
Git local authentication setup
Prepare the initial SAS Viya configuration in the local Git repository
Push the initial SAS Viya configuration to the Cloud Git Repository
Edit your code in the Google CSR
Introducing the Google Cloud Source Repositories
Google Cloud Source Repositories (or "Google CSR" to keep it short) is the Git Repository service available in the Google Cloud Platform market place*.
It has several interesting features (as you can see on the screenshot) and is a "fully-managed" Git repository.
Google CSR currently supports 3 types of local authentication :
- SSH : Consists of a private key that resides on your local system and a public key that you register with Google Cloud. When using this mode, you’ll have to use your private SSH key to authenticate yourself against the Git repository.
- Google Cloud CLI : Cloud Source Repositories lets you authenticate by using the google cloud CLI (gcloud). You need to run the gcloud init command to authenticate in your GCP project.
- Manually generated credentials, a script is generated in the Google portal to create the .gitconfig and .gitcookies files (using the Netscape/Mozilla cookie file format), which is passed in the in the Git HTTP session, when the git command is used.
Google also documents 3 distinct kind of URLs to use in order to "git clone" the repository, corresponding to the authentication systems :
- Cloning using SSH:
| git clone ssh://[EMAIL]@source.developers.google.com:2022/p/[PROJECT_ID]/r/[REPO_NAME] |
- Cloning using the gcloud command:
| gcloud source repos clone [REPO_NAME] --project=[PROJECT_NAME] |
- Cloning using the manually generated credentials:
When I started my research, I was not too sure which kind of local authentication to choose for my need to make the "sas-orchestration" tool read the SAS Viya configuration from the Google CSR.
However, with some hindsight, knowing that the sas-orchestration container relies on the "go-getter" library and requires that the "go-getter" protocol must be “Git over http(s)”, I could have been able to determine which authentication method to use…
(*) Note : Important: Effective June 17, 2024, Cloud Source Repositories isn't available to new customers. Organizations that have used Cloud Source Repositories prior to June 17, 2024 are not affected by this change.
Git local authentication setup
....
End of the suspense 😊
The authentication method to choose here is the "Manually generated credentials".
It is a choice by elimination: the “Git over SSH” protocol is not supported for the DaaS tools go-getter URLs and the gcloud CLI is not included in the sas-orchestration image.
When you opt for this method, you need to click on a link, which (after authentication with your Google account), generates a page as below:
This page gives you the snippets of code to run to configure your local git client (for Windows and Linux). When you run this code locally, it creates an http.cookiefile and references it in the Git configuration (~/.gitconfig file).
In a Linux operating system, it looks like this:
This is how Git can authenticate you over http(s) to the source.developers.google.com domain.
Once it is in place, you can run your git clone command to pull the content from your Google CSR.
Cloud Source Repository setup
It is very easy to create your first CSR repository in Google, you can use the web interface (go to https://source.cloud.google.com/repos and click on "Add repository") or the gcloud CLI with a command like:
Prepare the initial SAS Viya configuration in the local Git repository.
| gcloud source repos create raphpoumar-source-repo |
Now, imagine that you have your initial set of SAS Viya configuration files, locally available on your jumphost/bastion linux machine, and you want to move them to your brand-new Google Cloud Source Repo.
The following actions need to happen:
- Clone the empty CSR repository with git
Example:
It creates a local folder on your machine with the same name as your CSR repository. This new folder contains the git database (under the .git folder) is your local Git Repository.
- Setup the local Git configuration (~/.gitconfig).
Example:
|
#Setup git default identity git config --global user.email $EMAIL git config --global user.name $STUDENT git config --global credential.helper store git config --global http.sslVerify false |
- Create a local sub directory inside the local git directory and place your SAS Viya configuration files there.
It would look like this:
- Now add and commit the SAS Viya configuration files in the local git repository.
Note that the commit message appears in the Google CSR version history after the push.
At this point, the Git repository has been initiated but it remains local.
Push the initial SAS Viya configuration to the Cloud Git Repository.
- First, we add our local repository as a remote:
|
STUDENT=$(cat ~/student.txt) PROJECT_NAME=sas-gelsandbox git remote add googlehttps://source.developers.google.com/p/$PROJECT_NAME/r/$REPO_NAME |
- And then we can push from your local Git repository to the Google Cloud repository
But then…you might see this kind of message:
With a worrying line at the end saying that there was an error to push our files 🙁
Now, if we run the "git describe" command provided in this message, we see that the culprit is the file site-config/postgres/ServiceAccountKey.json.
What the message actually means, is that our push has been rejected because the service has detected that we were trying to upload what looks like a private key (the "ServiceAccountKey.json" file) !
In a standard usage of the Git repository, it is indeed a bad idea. We usually don't want to place any credential information in a code repository.
However, our case is a bit specific as we are not keeping source code but configuration files. In addition, we need this key to configure PostgreSQL and our Code Source repository is secured and private, so we can override this default security setting with the nokeycheck option.
So, let’s try again
| git push -o nokeycheck --all google |
Now we should see something like that:
The problem is fixed 😊
If we open the Cloud Source Repositories web page, we can confirm that our configuration files have been transferred !
Edit your configuration files in the Google CSR
OK so now my kustomization.yaml and site-config folder files are in the Google source repository, but wait…how do I update them if I want to change my SAS Viya configuration?
There are several ways. You can make the changes in your local git repository (from the local Linux machine used to do the first push) and add/commit/push them into the Google CSR remote repository.
But it is also possible to edit directly your configuration files from the Cloud Source Repositories web UI. You’ll first need to give additional IAM permissions in GCP (like "Source Repository Writer"), but then you can click on the "Edit code" button and a new Cloud shell instance will be started and the editor directly opens your configuration files (as shown below).
Finally, even easier, if you have Visual Studio Code (with the Git plugin) installed on your local Windows workstation, you can also use it to modify your SAS Viya configuration files, commit and push the changes (as long as you have updated your Windows Git configuration to use the manually created credentials).
So, now that our configuration files are in place and can be edited in the Google CSR, let’s see how to pull them when we deploy the software 😊
Deploying SAS Viya
The final step is to set up the go-getter URL in the sas-orchestration command and run it to deploy SAS Viya! The parameter that tells the sas-orchestration container where to find the SAS Viya Configuration file is "user-content".
Unlike with the GitLab example shown in my previous post, we cannot use a Git private token for the authentication (this authentication method was not available with the Google CSR). Instead, we need a way to make our credentials available for the sas-orchestration container.
Here is how the command looks like:
When we activate the verbose mode of the sas-orchestration tool we can see that the git clone command is executed inside the container to clone the remote repository.
So, as shown in the command example, the trick is to mount our .gitcookies and .gitconfig files inside the sas-orchestration container and to help Git to pick up the .gitconfig file by setting the $HOME environment variable. In this way the sas-orchestration tool has all the required credentials to perform the “git over http(s)” request to pull the viyaorch content from the Google repository.
Conclusion
Finally, we have seen that it is possible to run a Viya deployment from a configuration stored in the Google Cloud Source Repository. If you want to test it by yourself, a hands-on called "Deploy Viya 4 from Google Cloud Source Repositories" is available in the "course exercises" notes of the “SAS Viya 4: Deployment on Google Kubernetes Engine” GEL Lab: we’ll drive you through all the configuration steps described in this post.
We could then imagine to use the Google CSR as the central location for all our SAS Viya environments configurations.
As we've seen, the configuration files could be updated there in various ways and scenarios:
- Directly edited from the Google CSR UI
- Spin up an ad hoc VM that "git clone" the repository, update a given configuration, then push the changes. Then another instance or process can run the deployments.
- Synchronized from GitHub or Bitbucket repositories
The initial diagram was updated to reflect all these new opportunities 😊
Thanks for reading !
Find more articles from SAS Global Enablement and Learning here.
