Advanced Troubleshooting of Direct Marketing Agent Issues

[#1]

In the article, Managing the Direct Marketing Agent, we discussed common issues that you might encounter with the Direct Marketing agent and explored some helpful log messages that occur in the onprem_direct log file. However, what happens when the methods mentioned in that article fail to resolve the issue? You might encounter more advanced and complex issues when using the Direct Marketing agent that require advanced troubleshooting techniques.

In this follow-up article, we explore next-level troubleshooting techniques that can help you identify and resolve complex Direct Marketing agent problems. From checking your on-premises environment to analyzing system logs, we provide insights into methods that will help you identify the possible root cause of these issues and explain how to get your Direct Marketing agent to run smoothly.

Java Errors

When the platform receives updates to the Java environment, but the Java locations are not updated, the Direct Marketing agent might be unable to run.

Complete the following workarounds in any order to help debug the Java issues:

• Verify how the daemon is started for the Direct Marketing agent. This is determined during the installation of the Direct Marketing agent. For example, the way you are starting the agent might not be the way the administrator configured it.

A daemon (also known as background processes) is a Linux program that runs continuously in the background until you stop it.

Use the following commands to start the Direct Marketing agent:

To start the Direct Marketing agent as a service:  systemctl or upstart

or

Run the run.sh command

• Check the direct-agent-service.env file in the systemd directory for JAVA_HOME location

JAVA_HOME is typically set in the direct-agent-service.env file. You can issue a cat command to see the contents of the file. This command enables you to see whether that location is still good, as shown in the following example:

cd /opt/sas/onprem/systemd

Note: Your onprem directory might be different.

cat direct-agent-service.env

After issuing the command, the following is displayed:

#set JAVA_HOME to root of the Java 1.8 installation on this machine
       JAVA_HOME=/opt/sas/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre

Verify that the Java location is still valid and modify the path if it has changed.

• Make sure that when the Direct Marketing agent is set up, you are not starting with only the run.sh command. If you start the agent with only the run.sh command , when a user logs off from that session, the Direct Marketing agent is no longer running. To avoid this issue, start the Direct Marketing agent as a service with the systemctl or upstart commands.

You might see the following errors after operating system patching, software upgrades or when software changes are made, especially to Java,  when trying to start the OnPrem Agent (for example using the ./run.sh command):

Unable to start SAS_CI_360_OnpremAgent agent from xxxxxx server.

ERROR: JAVA_HOME is set to an invalid directory: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el7_9.x86_64

For more information, see the Direct Access Points documentation.

• Issue the following command, which contains general messages and system-related information to gather more information: sudo cat /var/log/messages

For example: you might see "ERROR_TENANT_MISSING". With this hint you could go and check the cionprem.properties file to verify the tenant information.

• Issue the following command: cat .bashrc (which is the profile to view the JAVA_HOME location).

The .bashrc file is a script file that executes when a user logs in.

• Contact your team to determine any recent changes performed on the system. An insignificant change (such as Java being recently updated, permission changes, and so on) can be very important and will help you in troubleshooting.
• From the onprem directory, locate the log’s directory and find the corresponding log to review possible errors.
  • onprem_direct.log.<date>
• At the Linux command prompt, enter the following one at a time, and note the results:

env

java -version

cd /usr/lib/jvm

ls -latr

• In rare cases, you might need to run the run.sh command in the onprem directory if that is where JAVA_HOME is set.

Note: We do not recommend that you put JAVA_HOME in the run.sh file. The JAVA_HOME setting can get removed when you upgrade the agent because it is not carried over into the new run.sh file.

•  Make sure to compare your old cionprem.properties file to that of the newly downloaded file to ensure that the information is the same as it was previously. You should always make a backup of the old cionprem.properties file before installing a new Direct Marketing agent or upgrading the Direct Marketing agent

System Issues or Changes

The Direct Marketing agent might be affected when you make system changes or issues. For example: If patching performed on the system and the SAS services were not stopped before-hand, you might see one of the following in the onprem-direct.log: 

ERROR [localhost-startStop-1] SID[] USER[] CC[] [] pringframework.web.context.ContextLoader - Context initialization failed
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'environmentProperties'
Caused by: com.atomikos.jdbc.AtomikosSQLException: Failed to grow the connection pool
Caused by: com.atomikos.datasource.pool.CreateConnectionException: Could not create JDBC connection
Caused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
Caused by: java.net.ConnectException: Connection refused (Connection refused)

WARN [localhost-startStop-1] SID[] USER[] CC[] [] rt.AnnotationConfigWebApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'environment' defined in ServletContext resource [/WEB-INF/spring-config/webapp-config.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public java.lang.String com.sas.analytics.crm.spring.client.SASEnvironmentFactory.getServiceRegistry() throws com.sas.analytics.crm.error.client.UserException] threw exception; nested exception is com.sas.analytics.crm.error.client.UserException: Bad Configuration

At this point you would want to check the SAS services and if necessary, restart all of the SAS Services:

To troubleshoot these issues, try the following tips:

• If the agent does not start and you are not sure why, check when you shut down the SAS Services (on-premises) last to see whether they stopped the Direct Agent too. It is important that—after restarting the SAS Services (on-premises)—you wait before starting the Direct Marketing Agent. We recommend waiting 30 minutes because the middle-tier usually takes up to 30 minutes to fully restart.
• We recommend that you restart SASServer1 (and any other SAS Servers that depend on SASServer1) if you make any changes to items such as users or application objects that are created in SAS metadata and are used by the agent. Then, after everything is running again, restart the agent itself.

For more information, see the Direct Access Points documentation.

SAS® Pooled Workspace Server and SAS® Object Spawner Issues

If there are issues with the SAS® 9.4 environment (such as with the SAS Pooled Workspace Server and the SAS Object Spawner), you might encounter ConnectionResources errors in the Direct Marketing agent log.

In this scenario, the following error occurs when you restart the Direct Marketing agent:

INFO  [pool-18-thread-3] SID[] USER[sasci360] CC[] [] ics.crm.custdata.datapattern.MAIQService - Retrieved logical server: SASApp_CI360 - Logical Pooled Workspace Server

ERROR [pool-18-thread-3] SID[] USER[] CC[] [] sas.iquery.execution.ConnectionResources - ConnectionResources unthreaded getConnection() failed.

Check the onprem_direct.log file to see whether it is trying to make a connection to the SAS Pooled Workspace Server - SASApp_CI360, as shown in the following example:

INFO  [pool-18-thread-3] SID[] USER[sasci360] CC[] [] ics.crm.custdata.datapattern.MAIQService - Retrieved logical server: SASApp_CI360 - Logical Pooled Workspace Server

ERROR [pool-18-thread-3] SID[] USER[] CC[] [] sas.iquery.execution.ConnectionResources - ConnectionResources unthreaded getConnection() failed.

com.sas.services.connection.FatalConnectionFactoryException: Unable to create class instance.

    at com.sas.services.connection.ClusterEnvelope.getConnection(ClusterEnvelope.java:240)

    at com.sas.services.connection.AggregationKernel.doGetConnection(AggregationKernel.java:242)

    at com.sas.services.connection.ConnectionFactoryKernel.getConnection(ConnectionFactoryKernel.java:325)

    at com.sas.services.connection.ConnectionFactoryShell.getConnection(ConnectionFactoryShell.java:75)

    at com.sas.services.connection.platform.PlatformConnectionFactory.getConnectionMetadata(PlatformConnectionFactory.java:533)

To circumvent this issue, complete the following steps:

1. Log in to SAS® Management Console using the sasUserId that is defined in the cionprem.properties file, which is located in the onprem directory. (For example, /opt/sas/onprem/ed.) Then, validate the SAS Pooled Workspace Server. If the SAS Pooled Workspace Server does not validate,  complete the next step.
2. Restart the SAS Object Spawner, and then validate the SAS Pooled Workspace Server again. If the SAS Pooled Workspace Server validates correctly, restart the Direct Marketing agent.

Tenant Access

In SAS® Customer Intelligence 360, the Tenant Access is a new feature in release 23.02. However, if you use the feature incorrectly, issues with the Direct Marketing agent can occur

When these issues occur, the Direct Marketing agent is unable to connect after making changes to the Tenant Access, as shown in the following image:

You might encounter the following errors in the onprem_direct.log:

WARN  [WebSocketClient@xxxxxxxx] SID[] USER[] CC[] [] as.analytics.crm.agent.impl.CI360MAAgent - streamClosed - errorCode: GATEWAY_CONNECTION_FAILED, message: 1007: CIException:MKTCMN74284:[MKTCMN74284] ERROR_INVALID_SOURCE_IP_ADDRESS

ERROR [AgentRestart-xxxxxx] SID[] USER[] CC[] [] as.analytics.crm.agent.impl.CI360MAAgent - processAgentConfig - Cannot retrieve direct marketing configuration. exception message:org.eclipse.jetty.client.HttpResponseException: HTTP protocol violation: Authentication challenge without WWW-Authenticate header

To resolve this issue, check and correct the IP Addresses that you recently added to the Tenant Access interface

For more information, see the Add Tenant Access documentation.

“Failed to Decrypt” Errors

If SAS 9.4 is not working correctly, you can encounter failed to decrypt error messages.

When you run the run.sh command in the Direct Marketing agent, you might encounter the following errors:

[sas@sas94mtier CI360DirectAgent]$ ./run.sh
0 [main] ERROR com.sas.vfabrictcsvr.decoder.EncryptDecrypt  - Failed to decrypt the data
2 [main] ERROR com.sas.vfabrictcsvr.decoder.PropertyDecoder  - The value could not be decrypted

WARN  [rtStop-1] SID[          ] USER[] CC[] [] ebApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cachedConfigurer' defined in ServletContext resource [/WEB-INF/spring-config/infrastructure-config.xml]: Cannot resolve reference to bean 'environmentProperties' while setting bean property 'properties'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'environmentProperties': FactoryBean threw exception on object creation; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.atomikos.jdbc.AtomikosSQLException: Failed to grow the connection pool

Workaround

To circumvent this issue, first make sure that you can log on to any of the SAS 9.4 middle-tier applications. (For example, http://midtier:port:SASStudio.)

If you are not able to log on, then you need to investigate why the middle-tier is not working. SAS Note 55426,  “Locating the log files for the SAS 9.4 middle tier”, provides the location of the SAS 9.4 middle-tier logs to help with your troubleshooting. Once you resolve the middle-tier issue, starting the Direct Marketing agent again.