topic Re: Backups are failing since move to new server in Administration and Deployment

Backups are failing since move to new server

rstorch01 — Wed, 10 May 2023 14:18:50 GMT

We have migrated our SAS server from centos 6 to 7 and in the process changed the hostname. The previous name still resolves via dns. The backups have failed since then. The errors indicate what is wrong but I am not familiar at all with SAS and the manuals referenced are not helpful for a linux environment. It would be nice to know how to fix this at the command line. Here are the errors:

The backup failed Error Code: 655, Stack Trace: Primary metadata server is not reachable using SAS Deployment Agent. Please make
sure that the agent is running on it.

Primary tier is not set. Please make sure that primary metadata tier is not excluded.

<BackupErrors><BackupError><BackupErrorEntry><errorHeader><errorCode>671</errorCode><errorImpact>
None of the backup operations will be able to complete except sas-list-schedule.</errorImpact><errorClassification>ERROR
</errorClassification><errorName>ERROR_CODE_DEPLOYMENT_AGENT_COMMUNICATION_FAILURE</errorName><errorCategory>GENERAL
</errorCategory><errorSummary>SAS Deployment Agent communication failed.</errorSummary><dateTimeOfError>

Re: Backups are failing since move to new server

gwootton — Wed, 10 May 2023 14:25:49 GMT

Did you see this step in the documentation related to host name changes:

Deleting Files in Vault Directories after a Host Name Change
https://go.documentation.sas.com/doc/en/bicdc/9.4/bisag/p1ugntmt9ufvtpn1tux6i272hwzl.htm#n1rg1g3k8n9zfbn174z7hkg151js

Re: Backups are failing since move to new server

rstorch01 — Wed, 10 May 2023 17:34:46 GMT

Thank you. I have moved the files to other temporary directories in the same spot to test the backup. I know I saw errors saying that an agent service wasn't running but I haven't been able to find what that agent service is and how to start it manually. I also wonder if running a backup during working hours will cause problems.

Any information on these issues would be very helpful. Thanks again.

Re: Backups are failing since move to new server

gwootton — Wed, 10 May 2023 17:39:58 GMT

The agent being referenced in the error is the SAS Deployment Agent.

Starting and Stopping the SAS Deployment Agent
https://go.documentation.sas.com/doc/en/bicdc/9.4/bisag/p0unuqm5ft99abn1hbgkzp5rjwj9.htm

Re: Backups are failing since move to new server

rstorch01 — Wed, 10 May 2023 17:47:44 GMT

Yes, I'm sorry, I did see that but since I am using linux, I wanted to start using the command line. So, running /SAS-installation-directory/SASDeploymentManager/9.4/sasdm.sh will launch the agent/daemon so that it will be available for the next scheduled backup?

I'm sorry to keep asking so many questions but I am not familiar with SAS at all and wanted to be careful.

Re: Backups are failing since move to new server

gwootton — Wed, 10 May 2023 18:08:56 GMT

The SAS Deployment Manager task "Start Deployment Agent" will start the deployment agent. If you don't want to use the graphical interface you can add the -console option. I think behind the scenes it runs <SASHome>/SASDeploymentAgent/9.4/agent.sh.
You can also run a backup manually if you don't want to wait for the next scheduled backup.

Re: Backups are failing since move to new server

rstorch01 — Wed, 10 May 2023 18:44:27 GMT

Thank you for your help. The agent appear to be running now. I do not know how to run the backup manually so I will see how it goes this weekend.

Re: Backups are failing since move to new server

gwootton — Wed, 10 May 2023 19:09:53 GMT

Here's the documentation on running an unscheduled backup:

Running an Unscheduled Backup
https://go.documentation.sas.com/doc/en/bicdc/9.4/evbr/p1k7tz3zj7um0gn1qhxn6whuz0yk.htm

Re: Backups are failing since move to new server

rstorch01 — Wed, 10 May 2023 19:17:33 GMT

Thank you so much. Again, I am so sorry but I need the command line. Hopefully things will run okay this weekend.

Re: Backups are failing since move to new server

gwootton — Wed, 10 May 2023 19:26:23 GMT

That documentation mentions how to run it with the command line under "See Also":

See Also
You can also use the sas-backup command to run a backup. See “Deployment Backup Command Reference” in the SAS Intelligence Platform: System Administration Guide.

Here is the link to the Deployment Backup Command Reference in the System Administration Guide:
https://go.documentation.sas.com/doc/en/bicdc/9.4/bisag/n0khpcet4rb0syn191i08w765cw1.htm

Re: Backups are failing since move to new server

rstorch01 — Thu, 11 May 2023 14:15:06 GMT

The backup program is still failing with these errors. The correct new hostname is showing in the error:

2023-05-11 00:30:00,124 [scheduler-1] WARN [sasadm@saspw] com.sas.svcs.backup.spi.UserConfigurationService - Ignoring update request for SASTiers as primary tier not found.

2023-05-11 00:30:00,234 [scheduler-1] ERROR [sasadm@saspw] com.sas.svcs.backup.spi.BackupServiceImpl - Primary tier is not
set. Please make sure that primary metadata tier is not excluded.

2023-05-11 00:30:00,243 [scheduler-1] ERROR [sasadm@saspw] com.sas.svcs.backup.aop.ErrorLoggingAspect - Error Start
||| <BackupErrors><BackupError><BackupErrorEntry><errorHeader><errorCode>671</errorCode><errorImpact>None of the backup operations will be able to complete except sas-list-schedule.</errorImpact><errorClassification>ERROR</errorClassification><errorName>ERROR_CODE_DEPLOYMENT_AGENT_COMMUNICATION_FAILURE</errorName><errorCategory>GENERAL</errorCategory><errorSummary>SAS Deployment Agent communication failed.</errorSummary><dateTimeOfError>2023-05-11T00_30_00</dateTimeOfError><logName>SASD
eploymentBackup9.4.log</logName><logFileLocation>/opt/sas/config/Lev1/Web/Logs/SASServer1_1</logFileLocation><operationType
>gethistory</operationType><errorOriginatingTier>newhostname.chmcres.cchmc.org</errorOriginatingTier>

I looked to see if the agent was actually running and see this process. I am not sure if this is the correct one. However, it is using the old hostname. I am not sure if this conflict is the problem but do not know how to change it in the agent:

$ ps -ef | grep DeploymentAgent
sas 4446 1 5 May04 ? 09:28:39 /opt/sas/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java -Dnop -Djava.util.logging.manager=com.springsource.tcserver.serviceability.logging.TcServerLogManager -Xmx4096m -Xss256k -Xms1024m -XX:+UseG1GC -XX:MaxGCPauseMillis=500 -XX:+DisableExplicitGC -XX:ReservedCodeCacheSize=96m -Djava.awt.headless=true -Dcom.sas.services.logging.disableRemoteList=true -Dcom.sas.services.logging.disableRemoteLogging=true -Dcom.sas.log.config.ignoreContextClassLoader=true -Dwebreportstudio.file.cleanup.interval=60 -Dspring.security.strategy=MODE_INHERITABLETHREADLOCAL -Dcom.sas.log.config.url=file:////opt/sas/config/Lev1/Web/Common/LogConfig -Dmulticast_udp_ip_ttl=1 -Djava.net.preferIPv4Stack=false -Djava.net.preferIPv6Addresses=false -Dmulticast.address=239.200.42.54 -Dmulticast.port=8561 -Dsas.jms.authentication.decorator=false -Dsas.scs.host=oldhostname.chmcres.cchmc.org -Dsas.scs.repository.dir=/opt/sas/config/Lev1/AppData/SASContentServer/Repository -Dcom.sas.server.isclustered=true -Dsas.scs.cas.host=sas.research.cchmc.org -Dsas.scs.cas.port=443 -Dsas.scs.cas.scheme=https -Dsas.scs.svc.host=sas.resaerch.cchmc.org -Dsas.scs.svc.port=443 -Dsas.scs.svc.scheme=https -Dsas.scs.scheme=http -Dsas.auto.publish.protocol=http -Dsas.container.identifier=vfabrictcsvr -Dsas.cache.locators=oldhostname.chmcres.cchmc.org[41415] -Dgemfire.conserve-sockets=false -Dspring.profiles.active=locators -Dsas.ttfontsvert.install.dir=/opt/sas/SASHome/ReportFontsforClients/9.4 -Dsas.scs.port=8080 -Dnet.sf.ehcache.skipUpdateCheck=true -Dorg.terracotta.quartz.skipUpdateCheck=true -Dsas.auto.publish.host=oldhostname.chmcres.cchmc.org -Dsas.auto.publish.port=8080 -Dsas.appserver.instance.id=SASServer1_1_oldhostname.chmcres.cchmc.org -Dconfig.lev.web.appserver.logs.dir=/opt/sas/config/Lev1/Web/Logs/SASServer1_1 -Djava.security.auth.login.config=/opt/sas/config/Lev1/Web/WebAppServer/SASServer1_1/conf/jaas.config -Dsas.metadata.use.cluster.properties=true -Dsas.deploy.dir=/opt/sas/config/Lev1/Web/WebAppServer/SASServer1_1/sas_webapps -Dorg.apache.activemq.SERIALIZABLE_PACKAGES=java.lang,java.util,java.net,java.sql,java.math,org.apache.activemq,org.fusesource.hawtbuf,org.springframework.remoting,org.springframework.security,com.thoughtworks.xstream.mapper,com.sas,org.apache.commons.logging,org.jasig.cas.client.validation,org.jasig.cas.client.authentication,org.jasig.cas.client.proxy -Dsas.deployment.agent.client.config=/opt/sas/SASHome/SASRemoteDeploymentAgentClient/2.1/config/deployagtclt.properties -Dsas.app.repository.path=/opt/sas/SASHome/SASVersionedJarRepository/eclipse -Dsas.retry.internal.url=true -Dsas.web.html.cdps.use.internal.urls=true -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /opt/sas/SASHome/SASWebApplicationServer/9.4/tomcat-8.5.32.A.RELEASE/bin/tomcat-juli-adapters.jar:/opt/sas/config/Lev1/Web/WebAppServer/SASServer1_1/lib/log4j.jar:/opt/sas/config/Lev1/Web/WebAppServer/SASServer1_1/lib:/opt/sas/config/Lev1/Web/WebAppServer/SASServer1_1/conf:/opt/sas/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/../lib/tools.jar:/opt/sas/SASHome/SASWebApplicationServer/9.4/tomcat-8.5.32.A.RELEASE/bin/bootstrap.jar:/opt/sas/SASHome/SASWebApplicationServer/9.4/tomcat-8.5.32.A.RELEASE/bin/tomcat-juli.jar -Dcatalina.base=/opt/sas/config/Lev1/Web/WebAppServer/SASServer1_1 -Dcatalina.home=/opt/sas/SASHome/SASWebApplicationServer/9.4/tomcat-8.5.32.A.RELEASE -Djava.io.tmpdir=/opt/sas/config/Lev1/Web/WebAppServer/SASServer1_1/temp org.apache.catalina.startup.Bootstrap start
root 134093 144189 0 10:11 pts/0 00:00:00 grep --color=auto DeploymentAgent

Re: Backups are failing since move to new server

gwootton — Thu, 11 May 2023 14:17:40 GMT

You should engage with SAS Technical Support so they can help correct your Deployment Agent configuration and Deployment Backup Configuration.