Hello mates, I'm developing script deploying whole Kubernetes enviroment + SAS Viya4 locally on one machine. I've done lot of effort to make it work as desired. Finally script is close to be ready and of course like other my "hardcore tasks" to by shared with you.
I know that there is limited support from SAS on custom made clusters so I'm asking you - fellow community for help.
I know that there is limited support from SAS on custom made clusters so I'm asking you - fellow community for help.
The problem I'm facing is that 99% of my pods have "sas-consul-client secret not found", and of course I researched https://support.sas.com/kb/67/349.html , created specified psps and still .. doesn't work.
The biggest mystery is... that all the yamls in site-config, sas-bases, sas-orchestra which are used to deploy ... don't contain creating this secret. There are only SecRefs 😐
Ok, here's my environment:
1. Installed kubelet=$KUBEVERSION kubectl=$KUBEVERSION kubeadm=$KUBEVERSION where KUBEVERSION="1.21.5-00"
2. Turned swap off
3. Installed docker
4. Inited cluster with kubeadm init --pod-network-cidr=$NETWORKCIDR --apiserver-advertise-address=$NETWORKADDR where NETWORKCIDR="192.168.0.0/16" && NETWORKADDR="10.0.110.99"
5. Created PSPs :
- psp-privileged.yaml << EOF
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: privileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
spec:
privileged: true
allowPrivilegeEscalation: true
allowedCapabilities:
- '*'
volumes:
- '*'
hostNetwork: true
hostPorts:
- min: 0
max: 65535
hostIPC: true
hostPID: true
runAsUser:
rule: 'RunAsAny'
seLinux:
rule: 'RunAsAny'
supplementalGroups:
rule: 'RunAsAny'
fsGroup:
rule: 'RunAsAny'
EOF
psp-baseline.yaml << EOF
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: baseline
annotations:
# Optional: Allow the default AppArmor profile, requires setting the default.
apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'
apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'
seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
spec:
privileged: false
# The moby default capability set, minus NET_RAW
allowedCapabilities:
- 'CHOWN'
- 'DAC_OVERRIDE'
- 'FSETID'
- 'FOWNER'
- 'MKNOD'
- 'SETGID'
- 'SETUID'
- 'SETFCAP'
- 'SETPCAP'
- 'NET_BIND_SERVICE'
- 'SYS_CHROOT'
- 'KILL'
- 'AUDIT_WRITE'
# Allow all volume types except hostpath
volumes:
# 'core' volume types
- 'configMap'
- 'emptyDir'
- 'projected'
- 'secret'
- 'downwardAPI'
# Assume that ephemeral CSI drivers & persistentVolumes set up by the cluster admin are safe to use.
- 'csi'
- 'persistentVolumeClaim'
- 'ephemeral'
# Allow all other non-hostpath volume types.
- 'awsElasticBlockStore'
- 'azureDisk'
- 'azureFile'
- 'cephFS'
- 'cinder'
- 'fc'
- 'flexVolume'
- 'flocker'
- 'gcePersistentDisk'
- 'gitRepo'
- 'glusterfs'
- 'iscsi'
- 'nfs'
- 'photonPersistentDisk'
- 'portworxVolume'
- 'quobyte'
- 'rbd'
- 'scaleIO'
- 'storageos'
- 'vsphereVolume'
hostNetwork: false
hostIPC: false
hostPID: false
readOnlyRootFilesystem: false
runAsUser:
rule: 'RunAsAny'
seLinux:
# This policy assumes the nodes are using AppArmor rather than SELinux.
# The PSP SELinux API cannot express the SELinux Pod Security Standards,
# so if using SELinux, you must choose a more restrictive default.
rule: 'RunAsAny'
supplementalGroups:
rule: 'RunAsAny'
fsGroup:
rule: 'RunAsAny'
EOF
psp-restricted.yaml << EOF
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: restricted
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default,runtime/default'
apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'
apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'
spec:
privileged: false
# Required to prevent escalations to root.
allowPrivilegeEscalation: false
requiredDropCapabilities:
- ALL
# Allow core volume types.
volumes:
- 'configMap'
- 'emptyDir'
- 'projected'
- 'secret'
- 'downwardAPI'
# Assume that ephemeral CSI drivers & persistentVolumes set up by the cluster admin are safe to use.
- 'csi'
- 'persistentVolumeClaim'
- 'ephemeral'
hostNetwork: false
hostIPC: false
hostPID: false
runAsUser:
# Require the container to run without root privileges.
rule: 'MustRunAsNonRoot'
seLinux:
# This policy assumes the nodes are using AppArmor rather than SELinux.
rule: 'RunAsAny'
supplementalGroups:
rule: 'MustRunAs'
ranges:
# Forbid adding the root group.
- min: 1
max: 65535
fsGroup:
rule: 'MustRunAs'
ranges:
# Forbid adding the root group.
- min: 1
max: 65535
readOnlyRootFilesystem: false
EOF6. Installed Calico, Metallb, Ingress
7. Created NFS Server and NFS Subdir Provisioner
8. Downloaded and created ARK-Report (in attachment)
9. Installed cert-manager and custom CA autority
1. Created directory structure for sasoperator and sas deployment
2. Successfully deployed sasoperator
3. Created kustomization.yaml for site-deployment
namespace: {{ NAME-OF-NAMESPACE }}
resources:
- sas-bases/base
- sas-bases/overlays/cert-manager-issuer
- sas-bases/overlays/network/networking.k8s.io
- sas-bases/overlays/cas-server
- sas-bases/overlays/internal-postgres
# If your deployment contains programming-only offerings only, comment out the next line
- sas-bases/overlays/internal-elasticsearch
- sas-bases/overlays/update-checker
- sas-bases/overlays/cas-server/auto-resources
configurations:
- sas-bases/overlays/required/kustomizeconfig.yaml
transformers:
# If your deployment does not support privileged containers or if your deployment
# contains programming-only offerings, comment out the next line
- sas-bases/overlays/internal-elasticsearch/sysctl-transformer.yaml
- sas-bases/overlays/required/transformers.yaml
- site-config/security/cert-manager-provided-ingress-certificate.yaml
- sas-bases/overlays/cas-server/auto-resources/remove-resources.yaml
# If your deployment contains programming-only offerings only, comment out the next line
- sas-bases/overlays/internal-elasticsearch/internal-elasticsearch-transformer.yaml
# Mount information
# - site-config/{{ DIRECTORY-PATH }}/cas-add-host-mount.yaml
components:
- sas-bases/components/security/core/base/full-stack-tls
- sas-bases/components/security/network/networking.k8s.io/ingress/nginx.ingress.kubernetes.io/full-stack-tls
patches:
- path: site-config/storageclass.yaml
target:
kind: PersistentVolumeClaim
annotationSelector: sas.com/component-name in (sas-backup-job,sas-data-quality-services,sas-commonfiles,sas-cas-operator,sas-pyconfig)
# License information
# secretGenerator:
# - name: sas-license
# type: sas.com/license
# behavior: merge
# files:
# - SAS_LICENSE=license.jwt
configMapGenerator:
- name: ingress-input
behavior: merge
literals:
- INGRESS_HOST={{ NAME-OF-INGRESS-HOST }}
- name: sas-shared-config
behavior: merge
literals:
- SAS_SERVICES_URL=https://{{ NAME-OF-INGRESS-HOST }}:{{ PORT }}
# - SAS_URL_EXTERNAL_VIYA={{ EXTERNAL-PROXY-URL }}
EOFChanged values and finally kustomization.yaml looks like :
namespace: sasoperator
resources:
- sas-bases/base
- sas-bases/overlays/cert-manager-issuer
- sas-bases/overlays/network/networking.k8s.io
- sas-bases/overlays/cas-server
- sas-bases/overlays/internal-postgres
# If your deployment contains programming-only offerings only, comment out the next line
- sas-bases/overlays/internal-elasticsearch
- sas-bases/overlays/update-checker
configurations:
- sas-bases/overlays/required/kustomizeconfig.yaml
transformers:
# If your deployment does not support privileged containers or if your deployment
# contains programming-only offerings, comment out the next line
- sas-bases/overlays/internal-elasticsearch/sysctl-transformer.yaml
- sas-bases/overlays/required/transformers.yaml
- site-config/security/cert-manager-provided-ingress-certificate.yaml
# If your deployment contains programming-only offerings only, comment out the next line
- sas-bases/overlays/internal-elasticsearch/internal-elasticsearch-transformer.yaml
# Mount information
# - site-config/{{ DIRECTORY-PATH }}/cas-add-host-mount.yaml
components:
- sas-bases/components/security/core/base/full-stack-tls
- sas-bases/components/security/network/networking.k8s.io/ingress/nginx.ingress.kubernetes.io/full-stack-tls
patches:
- path: site-config/storageclass.yaml
target:
kind: PersistentVolumeClaim
annotationSelector: sas.com/component-name in (sas-backup-job,sas-data-quality-services,sas-commonfiles,sas-cas-operator,sas-pyconfig)
# License information
# secretGenerator:
# - name: sas-license
# type: sas.com/license
# behavior: merge
# files:
# - SAS_LICENSE=license.jwt
configMapGenerator:
- name: ingress-input
behavior: merge
literals:
- INGRESS_HOST=vmkub01.local
- name: sas-shared-config
behavior: merge
literals:
- SAS_SERVICES_URL=https://vmkub01.local:443
# - SAS_URL_EXTERNAL_VIYA={{ EXTERNAL-PROXY-URL }}Pulled ark
docker pull cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-orchestration:1.64.0-20211012.1634057996496
docker tag cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-orchestration:1.64.0-20211012.1634057996496 sas-orchestration
created .yaml deployment with ark
docker run --rm \
-v $(pwd):/tmp/files \
sas-orchestration \
create sas-deployment-cr \
--deployment-data /tmp/files/$CERTS \
--license /tmp/files/$LICENCE \
--user-content /tmp/files/deploy \
--cadence-name $CADENCE \
--cadence-version $CADENCEVERSION \
> viya4-sasdeployment.yaml
where
export LICENCE="SASViyaV4_9xxxx_0_stable_2021.1.6_license_2021-10-21T071701.jwt"
export CERTS="SASViyaV4_9xxxx_certs.zip"
export SASNAMESPACE="sasoperator"
export CADENCE="stable"
export CADENCEVERSION="2021.1.6"
and file looks like (without certs and license ofc):
---
apiVersion: v1
kind: Secret
metadata:
creationTimestamp: null
name: sas-viya
stringData:
cacert: |
-----BEGIN CERTIFICATE-----
xxx
-----END CERTIFICATE-----
cert: |
-----BEGIN RSA PRIVATE KEY-----
xxxx
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
xxx
-----END CERTIFICATE-----
license:
xxx
---
apiVersion: orchestration.sas.com/v1alpha1
kind: SASDeployment
metadata:
annotations:
operator.sas.com/checksum: ""
creationTimestamp: null
name: sas-viya
spec:
caCertificate:
secretKeyRef:
key: cacert
name: sas-viya
cadenceName: stable
cadenceVersion: 2021.1.6
clientCertificate:
secretKeyRef:
key: cert
name: sas-viya
license:
secretKeyRef:
key: license
name: sas-viya
repositoryWarehouse:
updatePolicy: Never
userContent:
files:
kustomization.yaml: |
namespace: sasoperator
resources:
- sas-bases/base
- sas-bases/overlays/cert-manager-issuer
- sas-bases/overlays/network/networking.k8s.io
- sas-bases/overlays/cas-server
- sas-bases/overlays/internal-postgres
# If your deployment contains programming-only offerings only, comment out the next line
- sas-bases/overlays/internal-elasticsearch
- sas-bases/overlays/update-checker
configurations:
- sas-bases/overlays/required/kustomizeconfig.yaml
transformers:
# If your deployment does not support privileged containers or if your deployment
# contains programming-only offerings, comment out the next line
- sas-bases/overlays/internal-elasticsearch/sysctl-transformer.yaml
- sas-bases/overlays/required/transformers.yaml
- site-config/security/cert-manager-provided-ingress-certificate.yaml
# If your deployment contains programming-only offerings only, comment out the next line
- sas-bases/overlays/internal-elasticsearch/internal-elasticsearch-transformer.yaml
# Mount information
# - site-config/{{ DIRECTORY-PATH }}/cas-add-host-mount.yaml
components:
- sas-bases/components/security/core/base/full-stack-tls
- sas-bases/components/security/network/networking.k8s.io/ingress/nginx.ingress.kubernetes.io/full-stack-tls
patches:
- path: site-config/storageclass.yaml
target:
kind: PersistentVolumeClaim
annotationSelector: sas.com/component-name in (sas-backup-job,sas-data-quality-services,sas-commonfiles,sas-cas-operator,sas-pyconfig)
# License information
# secretGenerator:
# - name: sas-license
# type: sas.com/license
# behavior: merge
# files:
# - SAS_LICENSE=license.jwt
configMapGenerator:
- name: ingress-input
behavior: merge
literals:
- INGRESS_HOST=vmkub01.local
- name: sas-shared-config
behavior: merge
literals:
- SAS_SERVICES_URL=https://vmkub01.local:443
# - SAS_URL_EXTERNAL_VIYA={{ EXTERNAL-PROXY-URL }}
sas-bases: ""
site-config/security/cert-manager-provided-ingress-certificate.yaml: "## Example PatchTransformer to patch the secret used by nginx ingress objects\n##\n## In the following code, the locations that require user specified values are indicated by a capitalized and\n## hyphenated name set off by curly braces and a space at each end. You should replace this token with the \n## actual value.\n## Replace the curly braces, interior spaces, and the variable name.\n## For instance, \"sas-viya-issuer\"\n## should be replaced with the name of the cert-manager issuer that will issue certificates used to make\n## TLS connections to the SAS Viya applications, such as sas-viya-issuer.\n## If you use the suggested example, the correct, final syntax would be:\n## value: sas-viya-issuer\n##\n##\n---\napiVersion: builtin\nkind: PatchTransformer\nmetadata:\n name: sas-cert-manager-ingress-annotation-transformer\npatch: |-\n - op: add\n path: /metadata/annotations/cert-manager.io~1issuer\n value: sas-viya-issuer # name of the cert-manager issuer that will supply the Ingress cert, such as sas-viya-issuer\ntarget:\n kind: Ingress\n name: .*"
site-config/storageclass.yaml: |
kind: RWXStorageClass
metadata:
name: wildcard
spec:
storageClassName: nfs-clientthen :
kubectl apply -f viya4-sasdeployment.yaml -n sasoperator
finally got :
NAME STATE CADENCENAME CADENCEVERSION CADENCERELEASE AGE
sas-viya SUCCEEDED stable 2021.1.6 20211029.1635519350329 7h8m
but:
sas-model-management-c894776d9-dsvww 0/1 Init:CreateContainerConfigError 0 7h53m
sas-model-manager-app-795848d4db-f9tx5 0/1 CreateContainerConfigError 0 7h53m
sas-model-publish-f4f9f5d7d-c7hrc 0/1 Init:CreateContainerConfigError 0 7h53m
sas-model-repository-86f4b6cb47-gwddw 0/1 CreateContainerConfigError 0 7h53m
sas-model-studio-app-5bd79dfdb-7kwxz 0/1 CreateContainerConfigError 0 7h53m
sas-natural-language-conversations-576674865d-wssnw 0/1 CreateContainerConfigError 0 7h53m
sas-natural-language-generation-5bc9f5b9-28cq9 0/1 CreateContainerConfigError 0 7h53m
sas-natural-language-understanding-64d776d46b-j9wkt 0/1 CreateContainerConfigError 0 7h53m
sas-notifications-554875b7d5-99446 0/1 Init:CreateContainerConfigError 0 7h53m
sas-office-addin-app-5fc7d68d96-v5rr7 0/1 CreateContainerConfigError 0 7h53m
sas-opendistro-operator-56f45fb488-lsqkw 0/1 CreateContainerConfigError 0 7h53m
sas-parse-execution-provider-54b6567f59-rrhwv 0/1 CreateContainerConfigError 0 7h53m
sas-preferences-656d9cd848-87njx 0/1 Init:CreateContainerConfigError 0 7h53m
sas-prepull-85c69b74c7-bhgsn 1/1 Running 0 7h54m
sas-projects-8955cf56f-v8tf4 0/1 CreateContainerConfigError 0 7h53m
sas-pyconfig-j2zgm 0/1 Pending 0 7h53m
sas-rabbitmq-server-0 0/1 Pending 0 7h53m
sas-rabbitmq-server-1 0/1 Pending 0 7h53m
sas-rabbitmq-server-2 0/1 Pending 0 7h53m
sas-readiness-78955dc49d-mg8fp 0/1 CreateContainerConfigError 0 7h53m
sas-report-distribution-6d6b55ddd5-67mq4 0/1 Init:CreateContainerConfigError 0 7h53m
sas-report-execution-7498cf5d86-rrlgz 0/1 CreateContainerConfigError 0 7h53m
sas-report-renderer-6fd7d8d5-fkjcd 0/1 CreateContainerConfigError 0 7h53m
sas-report-services-group-7d8564487d-txjmq 0/1 CreateContainerConfigError 0 7h53m
sas-scheduled-backup-job-27260700-fzmf9 0/2 Pending 0 5h56m
sas-scheduler-776d8686c9-v7jml 0/1 CreateContainerConfigError 0 7h53m
sas-score-definitions-6bf66dcfd-xn6gw 0/1 Init:CreateContainerConfigError 0 7h53m
sas-score-execution-8d96cf55b-twvqk 0/1 Pending 0 7h53m
and all the pods with errors are like this (describe pod) :
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 35m default-scheduler Successfully assigned sasoperator/sas-conversation-designer-app-646ff4697c-k2xcf to vmkub01
Normal Pulling 35m kubelet Pulling image "cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-certframe:3.20.7-20211015.1634318362435"
Normal Pulled 34m kubelet Successfully pulled image "cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-certframe:3.20.7-20211015.1634318362435" in 59.23941659s
Normal Created 34m kubelet Created container sas-certframe
Normal Started 34m kubelet Started container sas-certframe
Normal Pulling 34m kubelet Pulling image "cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-conversation-designer-app:2.9.1-20211013.1634156561133"
Normal Pulled 24m kubelet Successfully pulled image "cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-conversation-designer-app:2.9.1-20211013.1634156561133" in 10m17.950160092s
Warning Failed 22m (x11 over 24m) kubelet Error: secret "sas-consul-client" not found
Normal Pulled 19s (x110 over 24m) kubelet Container image "cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-conversation-designer-app:2.9.1-20211013.1634156561133" already present on machine
Please help me get rid of this (error ofc :D)
