Proxy Use Requirements If you want to connect the agent through a forwarding proxy, you need to set certain settings, which are located in the Install and Configure the Direct Agent section of the SAS® Customer Intelligence 360 Administration Guide.   If you need to connect the agent through a proxy, set the following properties in the cionprem.properties file. This file is located in the installation directory of the Direct Marketing Agent. (For example, C:\install\CI360Direct or /install/CI360Direct.)   proxyHost: The host name of the proxy server. proxyPort: The port number. The default value is 80. nonProxyHosts: A list of hosts that must be connected directly and bypass the proxy. The value is a list of patterns that are separated by |. The patterns can start or end with a * for wildcards. Any host that matches one of these patterns is connected directly instead of through a proxy. Note: Use the following three properties when the user has an authenticated proxy: proxyUser: The user login for the proxy server. proxyRealm: The protected area in the proxy where the login and password are valid. proxyPassword: The user password for the proxy server. Here is an example: proxyHost=foo.demo.sas.com proxyPort=8181 nonProxyHosts=localhost|fooburger.demo.sas.com proxyUser=saszsr proxyRealm =”Access to the staging site"   See the Install and Configure the Direct Agent section of the SAS® Customer Intelligence 360 Administration Guide for more information. Supported Proxy Authentication SAS supports the following proxy authentication:   Basic - The standard mechanism for SAS web products where users enter credentials in a logon dialog box. Digest - An authentication type that does not require the password to be transmitted. Instead, the client uses the MD5 hashing algorithm to create a hash and sends it to the Server. SAS does not support the following proxy authentication:   NT LAN Manager (NTLM) Allow Domains In addition, to communicate with SAS Customer Intelligence 360, you must indicate which specific domains to allow in your firewall depending on the action that you want to perform. The Task 6: Allow Domains section of the SAS® Customer Intelligence 360 Administration Guide lists the domains that need to be allowed.   It is important to include ALL the information for your regions and for the actions that you want to take. Make sure to scroll down the page and include all the actions that you need. Here is an example:    SAS® 360 Engage: Direct Segment downloads for only direct marketing. Segment uploads for only direct marketing. Static link node list uploads and downloads only for direct marketing. Issue Symptoms The following symptoms might occur when there are issues with the Direct Marketing Agent: TimeoutException errors getting sessions (sid is null) segment downloads and uploads for direct marketing do not work INVALID_PROXY_CALLBACK Errors and Messages onprem_direct.log Messages The following messages might occur in the onprem_direct.log: Event stream connection to gateway failed: java.util.concurrent.TimeoutException: Idle timeout expired:   errorCode: GATEWAY_CONNECTION_FAILED, message: java.util.concurrent.TimeoutException: Idle timeout expired: When traffic flows regularly over the WebSocket, the connection stays up. However, the connection has issues when there is an idle period. In this scenario, the gateway sends a "ping" every 60 seconds. In this idle case, the agent never receives the ping from the agent logs. As a result, when the gateway does not receive a response to the ping within 30 seconds, it closes the connection and also has an idle time-out. (Note that the agent does not get a close event either). Then, once the agent does not receive a message for 70 seconds, it closes its end of the connection and tries to reconnect. This message likely occurs because there is network equipment between this agent and the gateway that has an idle time-out shorter than ten minutes. That equipment closes the WebSocket connection when it reaches its idle time-out, which is hit because the gateway pings come only every 70 seconds when there is no other traffic to be sent. At that point, a new connection has to be made to get traffic to flow again. To circumvent this message, try to increase the firewall time-out to more than one minute. org.jasig.cas.client.validation.TicketValidationException: INVALID_PROXY_CALLBACK   This message occurs because the SAS Customer Intelligence OnPrem Direct Services is pointing to the wrong host name and needs to point to the fully qualified host name where the Direct Marketing Agent is running. Common Proxy Errors 407 Proxy Authentication Required   The client must authenticate with the proxy server before the request can be processed. This error can be the result of missing or incorrect credentials.   429 Too many Requests   The gateway returns this error when a user has made too many REST requests of a particular type within the current 30-second window. To circumvent this error, spread out the time of the requests. Other General Errors These errors can occur regardless of whether you do or do not use proxies:   400 Bad Request   This error occurs when the proxy server cannot understand the request, which can be the result of any of the following: Incorrect formatting or syntax errors, making them unreadable by the server. The URL contains invalid characters or is improperly formatted. The request is too large for the server to handle (large files or excessive data). There are corrupted or invalid cookies that the server rejects. An incorrect HTTP header (request or response) is improperly formatted or contains incorrect values. 403 Forbidden   This error occurs when access to resources is denied, which can be due to any of the following: blocked ports blacklisted sites restricted access policies 502 Bad Gateway   This error occurs when the gateway or proxy server receives an invalid response from an upstream server. This error indicates that the upstream server is reachable but returned an unexpected or invalid response, which can be due to any of the following: Server overloads – The server is overwhelmed with too many requests. Technical issues on the destination server or upstream server. The destination server is the server that the gateway or proxy is trying to communicate with to fulfill the client’s request. The server is unavailable due to maintenance or problems. Network connectivity issues occur between the gateway and destination server. A firewall blocks the communication between servers. The Domain Name System (DNS) prevents the gateway from finding the destination server. To circumvent 502 errors, try any of the following workarounds: Refresh the page. Clear the browser cache. Check the network connection. Try another browser. Ask a coworker to try  the same steps that you took on their system. Check the firewall logs. Review the firewall rules and make sure that the ports and IP addresses are allowed. Contact the website support team. 503 Service unavailable   The proxy server cannot access the required service, which can be due to any of the following: server maintenance overloads   504 Gateway Timeout   This error occurs when the gateway or proxy server does not receive a timely response from the upstream server. This error indicates that the upstream server is reachable but did not respond in time, which can be due to any of the following: server overload server downtime network issues firewall blocks DNS issues To circumvent 504 errors, try any of the following workarounds: Refresh the page, Try another browser, Ask a coworker to try the same steps that you took on their system. Restart the router or computer. Contact the website support team. Contact the network administrators. Troubleshooting The Direct Marketing Agent Was Working But No Longer Works If the Direct Marketing Agent worked previously but you encounter an issue where it no longer works, consider the following:   Has anything recently changed in the environment? For example, an upgrade of an agent, network changes, system updates, modifications to SAS® Management Console, and so on. Check the status 360 page to make sure that there are no known issues for the Execution of Direct Marketing Tasks and Segment Maps: https://status.360.sas.com/. For example: issues with Direct Agent connectivity issues with Direct Agent connectivity degraded performance incident status degraded performance components execution of Direct Marketing tasks and segment maps Verify that no one has modified the proxy settings recently (cionprem.properties). Also, check the credentials.properties if you connect through an authenticated proxy. The client secret should match what is defined in the Access Point in the SAS Customer Intelligence 360 user interface (UI). If you are using JSON Web Tokens (JWT), it is still recommended to match the clientSecret. From the onprem directory, change to the logs directory, get the corresponding log, and review for errors (onprem_direct.log.<date> ). Contact your network team to see if they are getting additional messages or are aware of any known issues or changes that might be causing issues. As a best practice, always make sure that you are using the latest version of the Direct Marketing Agent. Additional Troubleshooting A good way to troubleshoot is to run the following curl command on the server where the Direct Marketing Agent is installed. If the curl command works, it does not mean that all is well with the proxy server. However, it is a good place to start in order to understand where the issue might be. curl -U "user-id":"pwd" -v -x proxy-name:proxy-port "https://extapigwservice-prod.ci360.sas.com/marketingGateway/commons/ping" --proxy-anyauth   Note: The values used for "user-id", "pwd", “proxy-name”, and “proxy-port” need to be the same values found in the cionprem.properties or credentials.properties files. If the proxy server does not require authentication, remove “-U "user-id":"pwd"” from the curl command.   ... View more

The Direct Marketing Agent might be affected when you make system changes or have a platform issue. It is very important to note that the SAS ® 9.4 Intelligence Platform must be functioning properly in order for the Direct Agent to work. The Direct Agent is needed in order to run SAS ® 360 Engage: Direct.   You might encounter the following symptoms when this issue occurs: A segment map is not able to select counts. A segment map execution on the Orchestration tab remains at 0% progress. If you experience this behavior, check the onprem-direct.log for a Bad Configuration warning.   “Bad Configuration” Warning   Issues that can cause Bad Configuration warnings include the following:   You did not stop all SAS services before you performed a patching on the system. You performed a patching on the system and files were overwritten or removed. You applied a SAS hot fix and did not complete all of the steps. Issues with the SAS environment (such as SAS Web Applications) are no longer working. A previous shutdown of the environment left stray processes running, and the ports are still in use. You made configuration changes to the SAS server, and the Direct Agent’s configuration files do not contain the updates. Here is an example of the Bad Configuration warning in the onprem-direct.log:     WARN [localhost-startStop-1] SID[] USER[] CC[] [] rt.AnnotationConfigWebApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'environment' defined in ServletContext resource [/WEB-INF/spring-config/webapp-config.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public java.lang.String com.sas.analytics.crm.spring.client.SASEnvironmentFactory.getServiceRegistry() throws com.sas.analytics.crm.error.client.UserException] threw exception; nested exception is com.sas.analytics.crm.error.client.UserException: Bad Configuration   Troubleshooting   To troubleshoot these issues, complete the following steps:   Make sure that all the SAS Services are running. Linux ./sas.servers status Windows Open the Services Application (services.msc).   2. Validate the SAS environment.     You can use your Instructions.html to validate that SAS ® Intelligence Platform is working as expected.   Details about how to validate SAS Intelligence Platform are in the Instructions.html files located in the <sasconfig>/Lev1/Documents/Instructions.html or <sasconfig>/Lev1/Documents/Backups folders.   Example Validation Steps   Here is an example of the validation steps:   1) In SAS ®  Management Console, validate the following:   SAS ®  Content Server   Test using an unrestricted user account (such as sasadm@saspw).   If SAS Content Server is not working, the following message might occur:   Code 503: Service unavailable   SAS ® Pooled Workspace Server a) Test using an unrestricted user account (such as sasadm@saspw), an account that is a member of the SAS Administrators group, or a regular user account.   b) Then, validate the mid-tier.   c) Make sure that you can log on to the SAS ® Web Administration Console, which is part of the SASServer1_1 SAS Web Application Server instance. If you cannot log on, the following message might occur:   Service Unavailable The server is temporarily unavailable to service your request due to maintenance downtime or capacity problems. Please try again later   Example link: http://<your mid-tier>:7980/SASAdmin or https://<your mid-tier>:/8343/SASAdmin)   SAS ® Web Server   a) Click the URL and look for the following message: Congratulations! You have successfully set up SAS Web Server.   Here is an example link: http://<your mid-tier>:7980 or https://<your mid-tier>:/8343).   2) SAS Server configuration changes.     Check the date of the existing files to determine whether modifications were made to these four files in the following directory:   <sasconfig>/Lev1/Web/WebAppServer/SASServer1_1/bin   * server.xml * jaas.config * catalina.properties * secure.file (only if configured to use the secure.file)   a) Then, compare these files to the ones in the Direct Agent directory (<sas_install>/<CI360Direct>/).   If these files do not match, rename them to “original” in the <sas_install>/<CI360Direct>/ directory as follows: * server.xml.original * jaas.config.original * catalina.properties.original * secure.file.original (only if the file currently exists in the directory and the system was configured using the secure.file   b) Copy the files from the <sasconfig>/Lev1/Web/WebAppServer/SASServer1_1/bin directory to the <sas_install>/<CI360Direct>/ directory.   * server.xml * jaas.config * catalina.properties * secure.file (only if the file currently exists in the directory and the system was configured using the secure.file)   For more details about these files, see the Install and Configure the Direct Agent section of the SAS® Customer Intelligence 360 Administration Guide.   If you need the Direct Agent and SAS 9.4 running quickly, the following steps often work:   1) Stop all the services.   2) Clean up the logs.   3) Restart all the services in the proper order.   For more information, complete the following steps:   Stop the Direct Agent. (For example, use systemctl stop direct-agent-service or the shutdown.sh script.) If on a separate machine, stop the middle-tier (also known as web-tier) components first. (For example, stop SASServerX_Y – WebAppServer first, then httpd-Web Server, then Cache Locator, and then JMS Broker.) Then, stop the compute, and finally the metadata tier.   If SAS Intelligence Platform is all on one machine, you can use the sas.servers stop script from Linux. On Linux, use the following script to stop the SAS Services:      sas.servers status      sas.servers stop Delete, move, or archive to a different path the contents of the following directories:   <SAS-configuration-directory>/Lev1/Web/WebAppServer/SASServerX_Y/logs/*                In the directories above, the X and Y in SASServerX_Y represent the different servers that you might have installed           (that is SASServer1_1,  SASServer2_1, and so on). <SAS-configuration-directory>/Lev1/Web/WebAppServer/SASServerX_Y/temp/*               (These temporary files do not need to be backed up.)             In the directories above, the X and Y in SASServerX_Y represent the different servers that you might have installed            (that is SASServer1_1, SASServer2_1, and so on). <SAS-configuration-directory>/Lev1/Web/WebAppServer/SASServerX_Y/work/*              (These temporary files do not need to be backed up)              In the directories above, the X and Y in SASServerX_Y represent the different servers that you might have installed             (that is SASServer1_1, SASServer2_1, and so on).   <SAS-configuration-directory>/Lev1/Web/WebServer/logs/* <SAS-configuration-directory>/Lev1/Web/gemfire/instances/ins_41415/*.log <SAS-configuration-directory>/Lev1/Web/gemfire/instances/ins_41415/*.dat <SAS-configuration-directory>/Lev1/Web/activemq/data/*             Note: Delete only the content inside the data folder. If you delete the data folder, you will not be able to start the service.            All the files in <SAS-configuration-directory>//Lev1/Web/activemq/data/* will be regenerated after the restart. <SAS-configuration-directory>/Lev1/Web/Logs/SASServerX_Y/*                      In the directories above, the X and Y in SASServerX_Y represent the different servers that you might have installed                      (that is SASServer1_1, SASServer2_1, and so on). 4. Start the SAS Services beginning with the metadata tier, the compute tier, followed by the middle-tier (also known as web tier) components. If SAS Intelligence Platform is all on one machine, you can use the sas.servers start script:   On Linux, use the following script to start the SAS Services:                 sas.servers start 5. It is very important that—after restarting the SAS Services —you wait before starting the Direct Marketing Agent. SAS recommends waiting 30 minutes because the middle-tier usually takes up to 30 minutes to fully restart. Then, you can start the Direct Marketing Agent (for example, systemclt start direct-agent-service).   IMPORTANT TIP: Make sure you know how the daemon is started for the Direct Marketing Agent, which is determined during the installation of the Direct Marketing Agent. You need to start the Direct Agent the way the administrator or installer configured it.   Prevention   To help prevent issues with the SAS 9.4 environment, it is very important to set up a schedule to clean up the environment. See the following SAS Communities article for additional details: Performance-Type Issues That You Might Encounter with the Direct Marketing Agent   If the Direct Agent does not start and you are not sure why, check when you shut down the SAS Services (on-premises) last to determine whether they stopped the Direct Agent too. It is important that—after restarting the SAS Services—you wait before starting the Direct Marketing Agent. SAS recommends waiting 30 minutes because the middle-tier usually takes up to 30 minutes to fully restart.   SAS also recommends that you restart SASServer1 (and any other SAS Servers that depend on SASServer1) if you make any changes to items. These items include users or application objects that are created in the SAS metadata and are used by the agent. Then, after everything is running again, restart the agent itself.   Before restarting the SAS Services, you can search for any stray SAS processes that might still be running by using the grep command. Here is an example:   ps -ef | grep "sassrv" | grep "sasexe/sas"   For more information, see the Linux man pages for the ps and kill commands.   For more information, see the Direct Agent documentation. ... View more

When starting the Direct Agent, the following error might occur in the onprem-direct.log file before the agent attempts to start:   2023-04-17 09:38:06,096 ERROR [pool-18-thread-12] SID[] USER[] CC[] [] ics.crm.security.ejb.SecurityManagerBean - Problem logging in com.sas.svcs.cluster.httpinvoker.UnavailableServiceException: The system attempted to find and access a service for interface 'org.springframework.security.core.userdetails.UserDetailsService' for 2 hours and was unsuccessful.   The SAS services not being started in the correct order is the main cause of this error. For additional information, see Starting Servers in the Correct Order.   Note: For the Windows environment, make sure that you are running the Direct Agent as a service. For additional information, see Running the Direct Agent as a Service in Windows.   Workaround To resolve this error, complete the following steps, which are specific to your setup:     Stop the Direct Agent under services (default “CustomerIntelligence360Satellite” service name).    Stop all SAS services.   If you stop the servers using the list that is provided in Starting Servers in the Correct Order, make sure that you stop the services in the inverse order from the chart. SAS ® Web Infrastructure Platform Data Server and SAS ® Metadata Server should be stopped last.   Make sure that there are no SAS processes still running. You can check if there are SAS processes running under the Windows Task Manager. (For example, make sure there are no sas.exe processes still running.) From the Windows Task Manager, select the Command Line column in order to see the command that was used to launch the processes. Look for processes that are running under the sassrv user account and for sas.exe processes that are running and end them.   Start all the SAS services in the proper order. See Starting Servers in the Correct Order for additional information.   Wait 30 minutes to give the SAS web applications plenty of time to start up. After 30 minutes, start the Direct Agent (which contains the default “CustomerIntelligence360Satellite” service name).   Make note of any error that you might encounter. If the same errors occur immediately, validate . Details about how to validate SAS Platform are in the Instructions.html files located in the <sasconfig>/Lev1/Documents/Instructions.html or <sasconfig>/Lev1/Documents/Backups folders   An example of the validation steps,   In SAS ® Management Console, validate the following: the SAS ® Content Server the SAS Pooled Workspace Server   Then, validate the following in the mid-tier: Ensure that the starts. Make sure that you can log on to a web application (such as SAS ® Studio or SASAdmin). Validate the clients. Make sure that you can log on to SAS ® Information Map Studio and/or SAS ® Enterprise Guide ® .   Note: If you restart the server to restart the services, change the Direct Agent to start manually (default “CustomerIntelligence360Satellite” service name) before restarting the servers. Then, restart the Direct Agent service manually after 30 minutes to give the SAS web application services a chance to start up properly.                         ... View more

This article describes the different performance-type issues that you might encounter with the Direct Marketing Agent and the possible workarounds for those problems.   Performance concerns can include the following types of issues: errors when starting the Direct Agent, including warnings about bad configuration noticeable slowness in the environment inability to create a task or segment tasks that no longer work or run extremely slowly processes not responding tasks that stop responding or do not recover   Workarounds The following workarounds are an excellent defense and go a long way in preventive maintenance: Routinely stop all SAS services and clean up the logging. See the Creating Space on the Mid-Tier section for more details. To ensure that your environment is set for optimal performance, see the “Tune the Direct Agent”. These steps ensure that there are always some SAS pooled workspace instances available for processing. If you experience unacceptable delays due to start-up costs, consider tuning these parameters as needed for your workload and machine size. Check the space on your environments regularly. (SAS Temporary Files (-work) and MATables directories or filesystems) Make sure that you are current with the latest hot fixes for your software. The hot-fix blogs are a good reference to help with the hot-fix process and generating a report. The report provides a visual of what is needed on all tiers: The SAS Hot Fix Analysis, Download and Deployment (SASHFADD) Tool - SAS Users Understanding the SAS Hot Fix Analysis, Download and Deployment Tool Report - SAS Users Keep a list of changes in your environment and the load on your system. This list can help you determine whether those changes added to recent performance changes. When performance is degraded, check which jobs are running on the system to determine whether you need to adjust the time at which your jobs run. (For example, consider not running jobs during high-traffic times.)   Creating Space on the Mid-Tier To clean up so that you have more space on the mid-tier, complete the following steps:  Stop the Direct Agent (for example: use systemctl stop direct-agent-service or shutdown.sh script). Stop the middle-tier components first. (For example, stop SASServerX_Y – WebAppServer first, then httpd-Web Server, then Cache Locator, and then JMS Broker.) On Linux, use the following script to stop the SAS Services:      sas.servers status      sas.servers stop or sas.servers.mid Delete, move, or archive to a different path the content of the following directories: <SAS-configuration-directory>/Lev1/Web/WebAppServer/SASServerX_Y/temp/*  (these temporary files do not need to be backed up) <SAS-configuration-directory>/Lev1/Web/WebServer/logs/* <SAS-configuration-directory>/Lev1/Web/gemfire/instances/ins_41415/*.log <SAS-configuration-directory>/Lev1/Web/gemfire/instances/ins_41415/*.dat <SAS-configuration-directory>/Lev1/Web/gemfire/instances/ins_41415/ConfigDiskDir/* <SAS-configuration-directory>/Lev1/Web/activemq/data/*  Note: Delete only the content inside the data folder. If you delete the data folder, you will not be able to start the service. All the files in <SAS-configuration-directory>/Lev1/Web/activemq/data/* will be regenerated after the restart.   <SAS-configuration-directory>/Lev1/Web/Logs/SASServerX_Y/*  In the directories above, the X and Y in SASServerX_Y represent the different servers that you might have (that is SASServer1_1, SASServer2_1, and so on).   Start the following middle-tier components again:           sas.servers status           sas.servers start   See Using the sas.servers Script on UNIX or z/OS to Start or Stop All Servers for additional information.   It is very important that—after restarting the SAS Services (on-premises)—you wait before starting the Direct Marketing Agent. We recommend waiting 30 minutes because the middle-tier usually takes up to 30 minutes to fully restart. Then, you can start the Direct Marketing Agent (for example: systemclt start direct-agent-service)   Additional References For more information about troubleshooting the Direct Marketing Agent, see the following resources: Advanced Troubleshooting of Direct Marketing Agent Issues Managing the Direct Marketing Agent “Bad Configuration” Messages Occur in the onprem_direct.log for the Direct Marketing Agent  Microsoft Windows Best Practices for the Direct Marketing Agent  ... View more

In the article, Managing the Direct Marketing Agent, we discussed common issues that you might encounter with the Direct Marketing agent and explored some helpful log messages that occur in the onprem_direct log file. However, what happens when the methods mentioned in that article fail to resolve the issue? You might encounter more advanced and complex issues when using the Direct Marketing agent that require advanced troubleshooting techniques.   In this follow-up article, we explore next-level troubleshooting techniques that can help you identify and resolve complex Direct Marketing agent problems. From checking your on-premises environment to analyzing system logs, we provide insights into methods that will help you identify the possible root cause of these issues and explain how to get your Direct Marketing agent to run smoothly.   Java Errors When the platform receives updates to the Java environment, but the Java locations are not updated, the Direct Marketing agent might be unable to run. Complete the following workarounds in any order to help debug the Java issues: Verify how the daemon is started for the Direct Marketing agent. This is determined during the installation of the Direct Marketing agent. For example, the way you are starting the agent might not be the way the administrator configured it. A daemon (also known as background processes) is a Linux program that runs continuously in the background until you stop it. Use the following commands to start the Direct Marketing agent: To start the Direct Marketing agent as a service:  systemctl or upstart or Run the run.sh command Check the direct-agent-service.env file in the systemd directory for JAVA_HOME location JAVA_HOME is typically set in the direct-agent-service.env file. You can issue a cat command to see the contents of the file. This command enables you to see whether that location is still good, as shown in the following example: cd /opt/sas/onprem/systemd Note: Your onprem directory might be different. cat direct-agent-service.env After issuing the command, the following is displayed: #set JAVA_HOME to root of the Java 1.8 installation on this machine        JAVA_HOME=/opt/sas/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre Verify that the Java location is still valid and modify the path if it has changed. Make sure that when the Direct Marketing agent is set up, you are not starting with only the run.sh command. If you start the agent with only the run.sh command , when a user logs off from that session, the Direct Marketing agent is no longer running. To avoid this issue, start the Direct Marketing agent as a service with the systemctl or upstart commands. You might see the following errors after operating system patching, software upgrades or when software changes are made, especially to Java,  when trying to start the OnPrem Agent (for example using the ./run.sh command): Unable to start SAS_CI_360_OnpremAgent agent from xxxxxx server. ERROR: JAVA_HOME is set to an invalid directory: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el7_9.x86_64 For more information, see the Direct Access Points documentation. Issue the following command, which contains general messages and system-related information to gather more information: sudo cat /var/log/messages For example: you might see "ERROR_TENANT_MISSING". With this hint you could go and check the cionprem.properties file to verify the tenant information.   Issue the following command: cat .bashrc (which is the profile to view the JAVA_HOME location). The .bashrc file is a script file that executes when a user logs in. Contact your team to determine any recent changes performed on the system. An insignificant change (such as Java being recently updated, permission changes, and so on) can be very important and will help you in troubleshooting. From the onprem directory, locate the log’s directory and find the corresponding log to review possible errors. onprem_direct.log.<date> At the Linux command prompt, enter the following one at a time, and note the results: env java -version cd /usr/lib/jvm ls -latr In rare cases, you might need to run the run.sh command in the onprem directory if that is where JAVA_HOME is set. Note: We do not recommend that you put JAVA_HOME in the run.sh file. The JAVA_HOME setting can get removed when you upgrade the agent because it is not carried over into the new run.sh file.  Make sure to compare your old cionprem.properties file to that of the newly downloaded file to ensure that the information is the same as it was previously. You should always make a backup of the old cionprem.properties file before installing a new Direct Marketing agent or upgrading the Direct Marketing agent System Issues or Changes The Direct Marketing agent might be affected when you make system changes or issues. For example: If patching performed on the system and the SAS services were not stopped before-hand, you might see one of the following in the onprem-direct.log:    ERROR [localhost-startStop-1] SID[] USER[] CC[] [] pringframework.web.context.ContextLoader - Context initialization failed Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'environmentProperties' Caused by: com.atomikos.jdbc.AtomikosSQLException: Failed to grow the connection pool Caused by: com.atomikos.datasource.pool.CreateConnectionException: Could not create JDBC connection Caused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. Caused by: java.net.ConnectException: Connection refused (Connection refused)   WARN [localhost-startStop-1] SID[] USER[] CC[] [] rt.AnnotationConfigWebApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'environment' defined in ServletContext resource [/WEB-INF/spring-config/webapp-config.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public java.lang.String com.sas.analytics.crm.spring.client.SASEnvironmentFactory.getServiceRegistry() throws com.sas.analytics.crm.error.client.UserException] threw exception; nested exception is com.sas.analytics.crm.error.client.UserException: Bad Configuration   At this point you would want to check the SAS services and if necessary, restart all of the SAS Services: To troubleshoot these issues, try the following tips: If the agent does not start and you are not sure why, check when you shut down the SAS Services (on-premises) last to see whether they stopped the Direct Agent too. It is important that—after restarting the SAS Services (on-premises)—you wait before starting the Direct Marketing Agent. We recommend waiting 30 minutes because the middle-tier usually takes up to 30 minutes to fully restart. We recommend that you restart SASServer1 (and any other SAS Servers that depend on SASServer1) if you make any changes to items such as users or application objects that are created in SAS metadata and are used by the agent. Then, after everything is running again, restart the agent itself. For more information, see the Direct Access Points documentation. SAS® Pooled Workspace Server and SAS® Object Spawner Issues If there are issues with the SAS® 9.4 environment (such as with the SAS Pooled Workspace Server and the SAS Object Spawner), you might encounter ConnectionResources errors in the Direct Marketing agent log.   In this scenario, the following error occurs when you restart the Direct Marketing agent:   INFO  [pool-18-thread-3] SID[] USER[sasci360] CC[] [] ics.crm.custdata.datapattern.MAIQService - Retrieved logical server: SASApp_CI360 - Logical Pooled Workspace Server ERROR [pool-18-thread-3] SID[] USER[] CC[] [] sas.iquery.execution.ConnectionResources - ConnectionResources unthreaded getConnection() failed. Check the onprem_direct.log file to see whether it is trying to make a connection to the SAS Pooled Workspace Server - SASApp_CI360, as shown in the following example: INFO  [pool-18-thread-3] SID[] USER[sasci360] CC[] [] ics.crm.custdata.datapattern.MAIQService - Retrieved logical server: SASApp_CI360 - Logical Pooled Workspace Server ERROR [pool-18-thread-3] SID[] USER[] CC[] [] sas.iquery.execution.ConnectionResources - ConnectionResources unthreaded getConnection() failed. com.sas.services.connection.FatalConnectionFactoryException: Unable to create class instance.     at com.sas.services.connection.ClusterEnvelope.getConnection(ClusterEnvelope.java:240)     at com.sas.services.connection.AggregationKernel.doGetConnection(AggregationKernel.java:242)     at com.sas.services.connection.ConnectionFactoryKernel.getConnection(ConnectionFactoryKernel.java:325)     at com.sas.services.connection.ConnectionFactoryShell.getConnection(ConnectionFactoryShell.java:75)     at com.sas.services.connection.platform.PlatformConnectionFactory.getConnectionMetadata(PlatformConnectionFactory.java:533)   To circumvent this issue, complete the following steps: Log in to SAS® Management Console using the sasUserId that is defined in the cionprem.properties file, which is located in the onprem directory. (For example, /opt/sas/onprem/ed.) Then, validate the SAS Pooled Workspace Server. If the SAS Pooled Workspace Server does not validate,  complete the next step. Restart the SAS Object Spawner, and then validate the SAS Pooled Workspace Server again. If the SAS Pooled Workspace Server validates correctly, restart the Direct Marketing agent. Tenant Access In SAS® Customer Intelligence 360, the Tenant Access is a new feature in release 23.02. However, if you use the feature incorrectly, issues with the Direct Marketing agent can occur When these issues occur, the Direct Marketing agent is unable to connect after making changes to the Tenant Access, as shown in the following image:     You might encounter the following errors in the onprem_direct.log: WARN  [WebSocketClient@xxxxxxxx] SID[] USER[] CC[] [] as.analytics.crm.agent.impl.CI360MAAgent - streamClosed - errorCode: GATEWAY_CONNECTION_FAILED, message: 1007: CIException:MKTCMN74284:[MKTCMN74284] ERROR_INVALID_SOURCE_IP_ADDRESS ERROR [AgentRestart-xxxxxx] SID[] USER[] CC[] [] as.analytics.crm.agent.impl.CI360MAAgent - processAgentConfig - Cannot retrieve direct marketing configuration. exception message:org.eclipse.jetty.client.HttpResponseException: HTTP protocol violation: Authentication challenge without WWW-Authenticate header To resolve this issue, check and correct the IP Addresses that you recently added to the Tenant Access interface For more information, see the Add Tenant Access documentation.   “Failed to Decrypt” Errors If SAS 9.4 is not working correctly, you can encounter failed to decrypt error messages. When you run the run.sh command in the Direct Marketing agent, you might encounter the following errors: [sas@sas94mtier CI360DirectAgent]$ ./run.sh 0 [main] ERROR com.sas.vfabrictcsvr.decoder.EncryptDecrypt  - Failed to decrypt the data 2 [main] ERROR com.sas.vfabrictcsvr.decoder.PropertyDecoder  - The value could not be decrypted WARN  [rtStop-1] SID[          ] USER[] CC[] [] ebApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cachedConfigurer' defined in ServletContext resource [/WEB-INF/spring-config/infrastructure-config.xml]: Cannot resolve reference to bean 'environmentProperties' while setting bean property 'properties'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'environmentProperties': FactoryBean threw exception on object creation; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.atomikos.jdbc.AtomikosSQLException: Failed to grow the connection pool Workaround To circumvent this issue, first make sure that you can log on to any of the SAS 9.4 middle-tier applications. (For example, http://midtier:port:SASStudio.) If you are not able to log on, then you need to investigate why the middle-tier is not working. SAS Note 55426,  “Locating the log files for the SAS 9.4 middle tier”, provides the location of the SAS 9.4 middle-tier logs to help with your troubleshooting. Once you resolve the middle-tier issue, starting the Direct Marketing agent again.       ... View more

This article describes the different performance-type issues that you might encounter with SAS Marketing Automation and the possible workarounds for those problems.   Performance concerns can include the following types of issues: noticeable slowness in the environment inability to create a campaign tasks that no longer work or run extremely slowly users not being able to log on processes not responding no responses when you click something in the user interface (UI) tasks that stop responding or do not recover   Workarounds The following workarounds are an excellent defense and go a long way in preventive maintenance:   Routinely stop all SAS services and clean up the logging. See the Creating Space on the Mid-Tier section for more details. To make sure that your environment is set for optimal performance, see SAS Note 60773, “Configuring SAS ® Pooled Workspace and SAS ® Stored Process servers for optimal performance with SAS ® Customer Intelligence.” To understand the settings that are used to optimally configure the SAS Pooled Workspace Servers and SAS Stored Process Servers for use with SAS® Customer Intelligence, see SAS SAS Note 40567, “Troubleshooting and tuning SAS ® Stored Process Server and SAS ® Pooled Workspace Server load balancing for SAS ® Marketing Automation releases.” Check the space on your environments regularly. Make sure that you are current with the latest hot fixes for your software—not just the ones for SAS Customer Intelligence. The hot-fix blogs are a good reference to help with the hot-fix process and generating a report. The report provides a visual of what is needed on all tiers: The SAS Hot Fix Analysis, Download and Deployment (SASHFADD) Tool - SAS Users Understanding the SAS Hot Fix Analysis, Download and Deployment Tool Report - SAS Users When performance is degraded, check which jobs are running on the system to determine whether you need to adjust the time that your jobs run. (For example, consider not running jobs during high-traffic times.) Make sure that when you stop services, there are no stray processes left behind. See SAS Note 57013, “Load-balancing problems can occur if a SAS ® Stored Process Server process fails to start” for more details. Keep a list of changes in your environment and the load on your system. This list can help you determine whether those changes added to recent performance changes. Creating Space on the Mid-Tier To clean up so that you have more space on the mid-tier, complete the following steps:   Stop the middle-tier components first. (For example, stop SASServerX_Y – WebAppServer first, then httpd-Web Server, then Cache Locator, and then JMS Broker.)   On Linux, use the following script to stop the SAS Services:      sas.servers status      sas.servers stop   Delete, move, or archive to a different path, the content of the following directories:   <SAS-configuration-directory>/Lev1/Web/WebAppServer/SASServerX_Y/temp/*  (these temporary files do not need to be backed up) <SAS-configuration-directory>/Lev1/Web/WebServer/logs/* <SAS-configuration-directory>/Lev1/Web/gemfire/instances/ins_41415/*.log, *.dat <SAS-configuration-directory>/Lev1/Web/activemq/data/*     Note: Delete only the content inside the data folder. If you delete the "data folder," you will not be able to start the service.   All the files you see in <SAS-configuration-directory>//Lev1/Web/activemq/data/* will be regenerated after the restart.   <SAS-configuration-directory>/Lev1/Web/Logs/SASServerX_Y/*   In the directories above, the X and Y in SASServerX_Y represent the different servers that you might have (that is SASServer1_1, SASServer6_1, and so on).   Start the following middle-tier components again:           sas.servers status           sas.servers start See Using the sas.servers Script on UNIX or z/OS to Start or Stop All Servers for additional information.     Additional Debugging Note that this section is not for regular errors that you might see in the GUI.   To determine whether additional debugging is necessary, consider the following questions: Have issues started taking a longer amount of time to run over time? Are you restarting the middle-tier services more often than normal due to a performance issue? Are you experiencing continued slowness that the cleanup from the previous section does not circumvent?   If you still experience issues similar to the ones above, enable garbage collection on SASServer1_1 and SASServer6_1 by following the instructions listed in https://support.sas.com/kb/59/508.html , “Enabling garbage-collection logging for SAS® 9.4 Environments.”   Note: When you enable garbage collection, the size of the logs increases. Before enabling garbage collection, make sure that you have enough space.   Once you enable garbage collection, run a campaign and reproduce the performance concern. Then, open a track with SAS Technical Support and provide the following information: the setenv.sh file for SASServer1_1 and SASServer6_1 on the middle-tier machine in the following locations: <SAS-configuration-directory>/Lev1/Web/WebAppServer/SASServer1_1/bin  <SAS-configuration-directory>/Lev1/Web/WebAppServer/SASServer6_1/bin the SASCustIntelCore6.6.log file on the middle-tier machine in the following location: <SAS-configuration-directory>/Lev1/Web/Logs/SASServer6_1/   verbose GC logs for SASServer1 and SASServer6 on the middle-tier machine in the following location: <SAS-configuration-directory>/LevN\Web\Logs\gc*.log the name of the campaign that you ran along with the campaign code the time that the campaign started and the time that the campaign finished who ran the campaign         ... View more

SAS® Customer Intelligence 6.6 products (like SAS® Marketing Automation) are not supported on SAS 9.4M8 (TS1M8). Any organization that upgrades their SAS installation to SAS 9.4M8 with SAS Customer Intelligence as part of that environment will make their SAS Customer Intelligence system non-functional.   Organizations that use multiple SAS products with SAS Customer Intelligence on the same SAS 9.4 environment need to separate those application installations onto two (2) separate SAS 9.4 platforms (one for SAS 9.4M7 and another for SAS 9.4M8). This separation means that organizations have only the following two (2) options to move to SAS 9.4 M8:   Install a fresh, new, and separate SAS 9.4 M8 system for use by the non SAS Customer Intelligence applications or solutions that they want to be on SAS 9.4M8 and leave the old SAS 9.4M7 (TS1M7), SAS 9.4M6 (TS1M6), or SAS 9.4M5 (TS1M5) Marketing Automation system running so that SAS Customer Intelligence can continue to use that platform version.   Install a new SAS Customer Intelligence depot including SAS 9.4M7 on a new system such that SAS Customer Intelligence runs independently from their other SAS applications that they want to upgrade to SAS 9.4M8. They would need to perform a migration using the SAS Migration Utility (SMU) from the old system to the new system to move their assets. They could then unconfigure SAS Customer Intelligence on the original system and then upgrade that system to SAS 9.4M8.   In summary, the only approach for keeping SAS Customer Intelligence on SAS 9.4M7 but upgrading their other solutions or applications to SAS 9.4M8 is two (2) separate backend SAS installations: one on SAS 9.4M8 and one on SAS 9.4M7, SAS 9.4M6, or SAS 9.4M5 for SAS Customer Intelligence.   Read Installation Note 69838 for more information. ... View more

Good morning, Just wanted to update anyone who will be updating to SAS Customer Intelligence 6.6, that there is a hotfix you will need if you are using Oracle, Postgres or Amazon Redshift database(s).   Please see the following SAS Note for more details and available hotfixes http://support.sas.com/kb/66025.   Thanks and have a safe and wonderful day!   ... View more