Hi @StuartRogers, First of all, thank you very much for such a useful article! The flow diagrams and narrative really make clear what is going on inside. I am currently trying to troubleshoot a hybrid kerberos w/ protocol transition problem and have been trying to find the forthcoming configuration post you mention at the end of this article. I have built my kustomization by referring to previous GEL Kerberos articles but these have all been for much earlier versions of Viya and I'm aware how rapidly Viya is changing and improving. Did you post the follow-up article? I've had a look for all GEL-tagged posts and can't see anything which fits. If it's not too much trouble I will outline my scenario in case you have any thoughts? I'm deploying Viya 2023.3-LTS on OpenShift 4.10 w/ k8s 1.24.6. I've included the relevant snippets of my kustomization.yaml below. I believe I've obeyed the kustomization-entries-ordering requirements for any elements which are position sensitive. resources:
kerberos/http
kerberos/sas-servers
kerberos/cas-servers
transformers:
cas-enable-host.yaml
kerberos/sas-servers/sas-kerberos-job-tls.yaml
kerberos/sas-servers/sas-kerberos-deployment-tls.yaml
kerberos/sas-servers/sas-kerberos-tls-transformer.yaml
kerberos/sas-servers/sas-kerberos-direct.yaml (I understand from your posts I no longer need this so will remove on the next respin)
configMapGenerator:
- name: sas-cas-config
behavior: merge
literals:
- CASCLOUDNATIVE=0 I followed the steps for Unconstrained Delegation from your earlier article (https://communities.sas.com/t5/SAS-Communities-Library/SAS-Viya-2020-1-4-and-later-Kerberos-Delegation-Configuration/ta-p/741911) to create my configMap settings. Identities is configured to use LDAP to AD and I'm using userPrincipalName (switched from sAMAccountName because I wanted to avoiding relying on krb5.conf/default_realm to handle name extension to Kerberos principal) as the key user attribute. I've registered the HTTP SPN in AD (configured with the proper AES enctypes, set for unconstrained delegation, the SPN has an FQDN which resolves via a wildcard A record to the IP of my ingress) and provided the keytab/krb5.conf in the necessary configMaps above (http, cas-servers, sas-servers). My krb5.conf seems fully-formed: from a shell in the same environment I can set KRB5_TRACE and watch kinit/klist/kvno work successfully (my krb5.conf has debug logging enabled) for the HTTP SPN. I've applied SCCs as appropriate, and when Viya runs as LDAP-only it is ok so I'm pretty sure my base platform is good. None of the clients are domain-joined to the AD I'm using for Identities/Kerberos; users logon to sas-logon via LDAP using uPN which is authN via LDAP and this is successful. When trying to start a SAS Studio session things go wrong. sas-launcher instantiates the sas-compute-server-nnnn pod which completes its initContainers, but in the sas-krb5-proxy container the log reports a series of errors, and compute and doesn't finish connecting back to Studio. {"level":"info","version":1,"source":"sas-run-tool","messageKey":"sonder-log-icu.run.tool.waiting.for.service.finish.log","properties":{"logger":"tools/run","caller":"impl/executor.go:263"},"timeStamp":"2023-08-02T18:51:30.389086+00:00","message":"Service successfully launched. Waiting for service to finish."} Info: 2023/08/02 18:51:30 main: running with effective configuration: {"addrString":"127.0.0.1:55901","libPath":"/usr/lib64/libgssapi_krb5.so.2","credPath":"/tmp/","spn":"HTTP/viya.apps.ocp.DOMAIN.NAME","credentialsAuthDomain":"KerberosAuth"} Info: 2023/08/02 18:51:30 main: SASLogonURL: https://sas-logon-app/SASLogon/kerberos/ticketService?spn=HTTP/viya.apps.ocp.DOMAIN.NAME Info: 2023/08/02 18:51:30 main: loading RootCAs from: SSL_CERT_FILE=/security/trustedcerts.pem Info: 2023/08/02 18:51:30 GSSAPI Trace: Loading "/usr/lib64/libgssapi_krb5.so.2" Info: 2023/08/02 18:51:30 GSSAPI Trace: krb.Load() succeeded: Debug: 2023/08/02 18:51:30 GSSAPI Debug: In AcquireCredWithKinit() Debug: 2023/08/02 18:51:30 krbAuth: Authenticator.found spn from credential HTTP/viya.apps.ocp.DOMAIN.NAME@CORRECT.KERB.REALM.NAME Info: 2023/08/02 18:51:30 main: Starting HTTP server on 127.0.0.1:55901 Debug: 2023/08/02 18:51:30 WriteCcacheHandler(1): found credential of type *oauth.UserData Debug: 2023/08/02 18:51:30 WriteCcacheHandler(1): parameters from query: Debug: 2023/08/02 18:51:30 negotiateFromSasLogon: forwarding Oauth credentials to https://sas-logon-app/SASLogon/kerberos/ticketService?spn=HTTP/viya.apps.ocp.DOMAIN.NAME Debug: 2023/08/02 18:51:30 negotiateFromSasLogon: SASLogon returned content types: [text/html;charset=UTF-8] Debug: 2023/08/02 18:51:30 basicFromCredentialsService: checking KerberosAuth authDomain for credentials Debug: 2023/08/02 18:51:30 basicFromCredentialsService: sending to https://sas-credentials/credentials/domains/KerberosAuth/secrets Error: 2023/08/02 18:51:30 WriteCcacheHandler(2): failed to proxy oauth credentials to SASLogon 2023/08/02 18:51:30 http: superfluous response.WriteHeader call from gitlab.sas.com/convoy/krb5-proxy/middleware/auth.(*cleanAuthenticateHeadersWriter).WriteHeader (auth.go:130) Warn: 2023/08/02 18:51:30 WriteCcacheHandler(2): No kerberos credential found in WriteCcacheHandler function. Aside from the JSON data at the beginning, this stanza repeats ~10 times. The parts which stand out to me are: 1. The Oauth mentions: I'm not using this as I do LDAP for authN in sas-logon-manager, 2. Checking KerberosAuth auth domain: I didn't think this would be a factor unless I was using stored user credentials, which I'm not. It feels to me that I've ended up with a configuration which is trying to do Kerberos Hybrid Auth with User Credentials, rather than KHA with Protocol Transition. I suspect that my config is a bit outdated as it's based on your posts for Viya 2020.x so tomorrow I am going to refresh my kustomization in light of your other posts about the changes in 2023.03, but if there is anything obvious you can spot I would very much appreciate your help 🙂
... View more
