topic Re: Question about the SAS Mirror Manager behavior in Administration and Deployment

Question about the SAS Mirror Manager behavior

EyalGonen — Sun, 12 May 2024 15:14:34 GMT

Hi experts,

I am using SAS Mirror Manager and I am noticing something odd. I am using the "--deployment-assets" parameter to point to the deployment assets file downloaded from my.sas.com but it seems like SAS Mirror Manager is always downloading the latest release and not the release found in the deployment assets file.

According to the doc I expected it to download the same cadence and release found in the deployment assets file and not the latest release. I verified the downloaded release in the downloaded path:

../sas_repos/rel/stable/2024.04

I see there:

relFormatVersion":"0.0.1","name":"stable","version":"2024.04","latest":{"duList":"lod/stable/2024.04/20240511.1715441583768

But my deployment assets file name has a different release:

stable_2024.04_20240423.1713898974806

The command I am using is:

$MIRRORMGRPATH/mirrormgr mirror registry --path ${MIRRORPATH} --deployment-data ${ASSETSPATH}/${CERTSFILE} --deployment-assets ${ASSETSPATH}/${ASSETSFILE}

What am I missing here? Maybe I am looking in the wrong place.. where can I see/verify which release was downloaded by mirror manager?

Re: Question about the SAS Mirror Manager behavior

gwootton — Mon, 13 May 2024 14:00:01 GMT

Mirror Manager has a --release option you can use to specify the release to download. If this is not specified it will pull the most recent release.
The command I use is:

cadencerelease=$(grep -E 'Cadence.Release' sas-bases/checksums.txt | awk '{print $NF}')
...
mirrormgr mirror registry --destination "$mirror" --deployment-data "SASViyaV4_${order}_certs.zip" --username "$mirroruser" --password "$mirrorpass" --cadence "${cadence}-${version}" --release "${cadencerelease}" --workers 10

Re: Question about the SAS Mirror Manager behavior

EyalGonen — Mon, 13 May 2024 14:07:20 GMT

Hi @gwootton

According to the doc at https://go.documentation.sas.com/doc/en/itopscdc/v_051/dplyml0phy0dkr/n1h0rgtr10fpnfn1mg0s8fgfuof8.htm#p16kb5lucncmbnn1c3x4dvpunim8 it says:

you can use the -–deployment-assets flag to pass in the cadence and release automatically.

So according to this if I specify --deployment-assets (as I did) it should pick up the cadence and the release automatically from the file pointed by this parameter.

So maybe I am looking at the wrong place to verify which release it actually downloaded. Can you please help telling me how to identify which exact release was downloaded with mirror manager?

Re: Question about the SAS Mirror Manager behavior

gwootton — Mon, 13 May 2024 16:14:58 GMT

Sorry, you are correct that --deployment-assets should do the same thing. Also it seems the default behavior has changed so when no release is supplied it downloads all of them.
When I look at my sas-bases/lod/stable/2024.04/ contents the release there matches the release I specified in the release options, so it sounds like mirrormgr is not working as expected.
What version of mirror manager are you using? (mirrormgr --version)
Did you download it recently?

Re: Question about the SAS Mirror Manager behavior

EyalGonen — Mon, 13 May 2024 16:22:39 GMT

Hi @gwootton

My mirror manager version version is

mirrormgr:
version : v0.36.1
build date : 2023-12-08
git hash : a02be0f
go version : go1.21.4 X:boringcrypto
go compiler : gc
platform : linux/amd64

So what you are saying is that to verify which release was actually downloaded I need to look at "sas-repo/lod/stable/2024.04/ "? I should see there a folder with the release that was downloaded? This is the way to verify what was downloaded?

Re: Question about the SAS Mirror Manager behavior

gwootton — Mon, 13 May 2024 16:26:45 GMT

@EyalGonen yes, the directories in that path would be for the various releases that have been downloaded. From the file you mentioned initially, it seems to have the release within an JSON object called "latest", so it may be that this was not what was downloaded and just the details on that cadence and version.

Re: Question about the SAS Mirror Manager behavior

EyalGonen — Mon, 13 May 2024 16:30:50 GMT

Hi @gwootton

Initially I did not know where to look to understand what was actually downloaded so I looked in the wrong place. Now in the folder you mentioned I have two files: one with the correct version (same as deployment assets) and another which is with a bit newer version:

-rw-r--r--. 1 ec2-user ec2-user 6302589 May 13 14:51 20240423.1713898974806
-rw-r--r--. 1 ec2-user ec2-user 6303778 May 11 16:19 20240511.1715441583768

I am not sure what I did anymore 🙂

I think I will delete and re-download and see what happens. I will update. Thanks a lot!

Re: Question about the SAS Mirror Manager behavior

EyalGonen — Mon, 13 May 2024 17:01:38 GMT

So I think I messed things up 🙂

I just checked and if I use the --deployment-assets flag then mirror manager automatically selects and downloads the cadence and release specified in the deployment assets file.

I am not sure how I ended up with two downloaded releases before...

Thanks @gwootton for your help!

Re: Question about the SAS Mirror Manager behavior

EyalGonen — Tue, 21 May 2024 06:01:56 GMT

Hi @gwootton

One more general question about this subject: can I use mirror manager to download new releases of the same cadence version to the same download location I used to download earlier releases?

Same question this time about downloading difference version of the same cadence to the same folder (for example, stable 2024.04 and stable 2024.05).

Same question about different cadences (for example stable 2024.04 and lts 2023.10)

Thanks,

Eyal

Re: Question about the SAS Mirror Manager behavior

gwootton — Tue, 21 May 2024 15:01:00 GMT

@EyalGonen yes, you can use the same location, what you end up with is in the repo path under docker/v2/viya-4-x64_oci_linux_2-docker paths for each image and version associated with each release downloaded.

Re: Question about the SAS Mirror Manager behavior

EyalGonen — Mon, 26 Aug 2024 02:17:24 GMT

Hi @gwootton and all,

How can I use SAS Mirror Manager to remove a specific release that was uploaded to AWS ECR?

For example, let's say I upload to ECR three different SAS Viya releases Stable 2024.05, 2024.06, 2024.07 and I want to delete one of them from ECR.

Re: Question about the SAS Mirror Manager behavior

gwootton — Mon, 26 Aug 2024 12:48:39 GMT

@EyalGonen I don't believe SAS Mirror Manager has such a function.

Re: Question about the SAS Mirror Manager behavior

EyalGonen — Mon, 26 Aug 2024 13:21:00 GMT

Hi @gwootton

So how do you cleanup the ECR?

If I do multiple SAS Viya upgrades, each time I load the new release to ECR and it starts filling up with many images from different releases. Each release is ~80GB.

I found an option for SAS Mirror Manager called "--remove-old" but not sure if it is valid for ECR nor how to exactly use it. I am kind of afraid to run it as I am not sure what it does.

Please advise.

Re: Question about the SAS Mirror Manager behavior

gwootton — Mon, 26 Aug 2024 13:45:38 GMT

@EyalGonen I tested running with the --remove-old option and received this message:
Flag --remove-old has been deprecated, please use --remove-after-upload, and that option would only remove from the local repo after they've been uploaded, so I don't think that would do what you want.

It looks like ECR has lifecycle options you can configure:
https://docs.aws.amazon.com/AmazonECR/latest/userguide/LifecyclePolicies.html

It looks like one way you could do this is with the tagStatus of "tagged" and "imageCountMoreThan" countType, which sounds like it would tell the ECR to remove all tagged images older than the last n images pushed.

Re: Question about the SAS Mirror Manager behavior

JuanS_OCS — Mon, 26 Aug 2024 20:01:58 GMT

Silly answer but, besides the SAS tool or the policies you could still use the ECR CLI or webtool to delete images?

aws ecr batch-delete-image --repository-name <repository_name> --image-ids imageTag=<tag>

Re: Question about the SAS Mirror Manager behavior

EyalGonen — Tue, 27 Aug 2024 04:31:55 GMT

Hi @JuanS_OCS

How do you know which images to delete? For example, I want to delete all images in ECR that belong to SAS release Stable 2024.04. I am not aware of any image tag that associates an image with Stable 2024.04. Maybe I am wrong. Please let me know if you find how to do it.

Re: Question about the SAS Mirror Manager behavior

gwootton — Wed, 28 Aug 2024 17:09:20 GMT

If you have the assets for a given release, you could grep the overlays/required/transformers.yaml to identify the tag for each image associated with that release, for example:

$ grep newTag -m1 -B1 sas-bases/overlays/required/transformers.yaml
newName: cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-activities
newTag: 1.53.15-20221103.1667486852217

But the version can change during a release cycle, if an update is pushed to 2024.04 for a given image and you pulled the latest assets for it, then the image tag would be different for that image than what you might have in your ECR.

Using the lifecycle option lets you instruct the ECR to remove all entries that meet given criteria, while keeping some history.

Re: Question about the SAS Mirror Manager behavior

EyalGonen — Thu, 05 Sep 2024 06:06:12 GMT

Hi @gwootton

If I understand correctly the ECR lifecycle option is at the repository level and not at the registry level (correct me if I am wrong) so this means that I need to set it up for all the SAS repositories and there are dozens of them...

Has anyone tried this lifecycle approach in a production site?

I think the SAS Mirror Manager should really have a "cleanup" option since it can tell the exact image tags associated with each Cadence release and it can safely remove images from the CR.

Re: Question about the SAS Mirror Manager behavior

gwootton — Thu, 05 Sep 2024 17:10:31 GMT

I believe this is something done at the repository level, so it might be easier to use the CLI to loop through all your repositories to apply the lifecycle policy to each. I've not had any experience using the functionality in AWS. I don't know of any plans to add that functionality to Mirror Manager.

It looks like there is functionality in the AWS console to preview the results of the lifecycle policy before applying it.

https://docs.aws.amazon.com/AmazonECR/latest/userguide/lpp_creation.html