BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
HULK
Obsidian | Level 7

Hi!

 

Faced the following problem:

I tried to install and configure SAS 9.4.
The installation process runs without problems, but during the configuration process, the "Load Content" error occurs:

HULK_0-1599813504640.png

 

The error in the corresponding log is as follows:

 

loadContent:
Property "webinfpltfm.migration.is_enabled" has not been set
     [echo] user: sasadm@saspw
     [echo] contentLocation: /sas/sashome_ma/SASWebInfrastructurePlatform/9.4/Config/Deployment/Content
     [echo] targetUrl: https://my_hostname:8343/SASContentServer/repository/default/sasdav
[DAV Restore] Executing '/sas/sashome_ma/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java' with arguments:
[DAV Restore] '-Djava.system.class.loader=com.sas.app.AppClassLoader'
[DAV Restore] '-Dsas.app.launch.config=/sas/sashome_ma/SASWebInfrastructurePlatform/9.4/picklist'
[DAV Restore] '-Dsas.ext.config=/sas/sashome_ma/sas.java.ext.config'
[DAV Restore] '-classpath'
[DAV Restore] '/sas/sashome_ma/SASVersionedJarRepository/eclipse/plugins/sas.launcher.jar'
[DAV Restore] 'com.sas.services.webdav.WebDAVRestore'
[DAV Restore] '-url'
[DAV Restore] 'https://my_hostname:8343/SASContentServer/repository/default/sasdav'
[DAV Restore] '-user'
[DAV Restore] 'sasadm@saspw'
[DAV Restore] '-pwd'
[DAV Restore] '********'
[DAV Restore] '-replace'
[DAV Restore] '/sas/sashome_ma/SASWebInfrastructurePlatform/9.4/Config/Deployment/Content'
[DAV Restore]
[DAV Restore] The ' characters around the executable and arguments are
[DAV Restore] not part of the command.
[DAV Restore] Output redirected to property: loadContent.stdout
[DAV Restore] Java Result: 8
     [echo] WebDAVRestore (9.4.004)
     [echo] Current time is: 2020-09-03T12:52:15.781+0300
     [echo] Restoring to "/SASContentServer/repository/default/sasdav" for user "sasadm@saspw" from "/sas/sashome_ma/SASWebInfrastructurePlatform/9.4/Config/Depl
oyment/Content"
     [echo] com.sas.services.webdav.DAVException: 501 (Not Implemented) during PROPFIND on /SASContentServer/repository/default/sasdav
     [echo]     at com.sas.services.webdav.DAVResource.processError(DAVResource.java:3381)
     [echo]     at com.sas.services.webdav.DAVResource.getProperties(DAVResource.java:1152)
     [echo]     at com.sas.services.webdav.DAVResource.getProperties(DAVResource.java:1061)
     [echo]     at com.sas.services.webdav.WebDAVRestore.doResourceSource(WebDAVRestore.java:192)
     [echo]     at com.sas.services.webdav.WebDAVRestore.main(WebDAVRestore.java:1205)
     [echo] Return Code: 8
[propertyfile] Updating property file: /sas/sasconfig/Lev1/Logs/Configure/webinfpltfm_config_status.properties
[propertyfile] Updating property file: /sas/sasconfig/Lev1/Logs/Configure/webinfpltfm_config_status.properties
     [echo] Load Content failed for: SAS Web Infrastructure Platform

 

I have some suggestions about this, but I would like to hear the opinions of experts on this issue.

Maybe someone has already encountered a similar error?

1 ACCEPTED SOLUTION

Accepted Solutions
JuanS_OCS
Amethyst | Level 16

Hi @HULK ,

 

yes, your original issue happens because SASServer1_1, where your SAS Content Server is hosted, cannot start. And this happens because the Cache Locator (gemfire), cannot start.

 

When you are troubleshooting issues it is critical to assume nothing. I think it is still worth to check if network it is an issue. It could be a firewall issue, or something else. It is simple, the error below it is self-explanatory:

Unable to find a free port in the membership-port-range

Google it and you will get: Modify the Cache Locator configuration to accommodate a firewall https://documentation.sas.com/?docsetId=bimtag&docsetTarget=n1dvn134xe84k0n1j14o0k534kv5.htm&docsetV...

Pretty useful document.

 

I have a few questions: 

 

  • what is the maintenance level of this as 9.4? SAS 9.4M6? M7?
  • how many IPs/NiC has this server got?
  • Is any of them a public IP address?
  • do you have the server firewalls (firewalld/iptables/other) enabled?
  • on what IP are your web services (and gemfire) listening?

 

You should check as well https://support.sas.com/kb/52/670.html

 

 

 

 

 

 

View solution in original post

13 REPLIES 13
AnandVyas
Ammonite | Level 13

Have the services on compute and mid-tier started successfully and are currently running during the load?

HULK
Obsidian | Level 7

Some services start correctly, some don't.
For example, SASServer1_1 starts with the following error: 

 

2020-09-03 10:57:45,343 INFO  (localhost-startStop-3) [org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/SASWIPClientAccess]] Initializing Spring root WebAp
plicationContext
2020-09-03 10:58:00,173 ERROR (localhost-startStop-5) [org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/SASPreferences]] Exception sending context initiali
zed event to listener instance of class [com.sas.svcs.webapp.servlet.springframework.DefaultContextLoaderListener]
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.sas.svcs.cache.cacheableBeanPostProcessor' defined in class path resource [com
/sas/svcs/data/caches/gemfire/sas_gemfire_config.xml]: Cannot resolve reference to bean 'sas_gemfire_cache' while setting bean property 'cache'; nested exception is org.s
pringframework.beans.factory.BeanCreationException: Error creating bean with name 'sas_gemfire_cache': FactoryBean threw exception on object creation; nested exception is
 com.gemstone.gemfire.SystemConnectException: Unable to find a free port in the membership-port-range
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:334)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1419)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1160)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:195)
        at org.springframework.context.support.AbstractApplicationContext.registerBeanPostProcessors(AbstractApplicationContext.java:762)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:467)
        at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:410)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
        at com.sas.svcs.webapp.servlet.springframework.DefaultContextLoaderListener.contextInitialized(DefaultContextLoaderListener.java:44)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4792)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5256)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
        at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:629)
        at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1839)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sas_gemfire_cache': FactoryBean threw exception on object creation; nes
ted exception is com.gemstone.gemfire.SystemConnectException: Unable to find a free port in the membership-port-range
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:173)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:102)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1467)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:304)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)
        ... 28 more
Caused by: com.gemstone.gemfire.SystemConnectException: Unable to find a free port in the membership-port-range
        at com.gemstone.gemfire.internal.SocketCreator.createServerSocketUsingPortRange(SocketCreator.java:873)
        at com.gemstone.gemfire.internal.tcp.TCPConduit.createServerSocket(TCPConduit.java:397)
        at com.gemstone.gemfire.internal.tcp.TCPConduit.startAcceptor(TCPConduit.java:340)
        at com.gemstone.gemfire.internal.tcp.TCPConduit.<init>(TCPConduit.java:247)
        at com.gemstone.gemfire.distributed.internal.direct.DirectChannel.<init>(DirectChannel.java:151)
        at com.gemstone.gemfire.distributed.internal.membership.jgroup.JGroupMembershipManager.initialize(JGroupMembershipManager.java:1657)
        at com.gemstone.gemfire.distributed.internal.membership.jgroup.JGroupMemberFactory.newMembershipManager(JGroupMemberFactory.java:112)
        at com.gemstone.gemfire.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:89)
        at com.gemstone.gemfire.distributed.internal.DistributionManager.<init>(DistributionManager.java:1215)
        at com.gemstone.gemfire.distributed.internal.DistributionManager.create(DistributionManager.java:610)
        at com.gemstone.gemfire.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:623)
        at com.gemstone.gemfire.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:282)
        at com.gemstone.gemfire.distributed.DistributedSystem.connect(DistributedSystem.java:1651)
        at com.gemstone.gemfire.cache.CacheFactory.create(CacheFactory.java:227)
        at org.springframework.data.gemfire.CacheFactoryBean.createCache(CacheFactoryBean.java:407)
        at org.springframework.data.gemfire.CacheFactoryBean.init(CacheFactoryBean.java:279)
        at org.springframework.data.gemfire.CacheFactoryBean.getObject(CacheFactoryBean.java:451)
        at org.springframework.data.gemfire.CacheFactoryBean.getObject(CacheFactoryBean.java:76)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:166)
        ... 33 more
2020-09-03 10:58:00,174 ERROR (localhost-startStop-5) [org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/SASPreferences]] Exception sending context initialized event to listener instance of class [com.sas.svcs.webapp.servlet.FrameworkContextListener]
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.sas.svcs.cache.cacheableBeanPostProcessor' defined in class path resource [com/sas/svcs/data/caches/gemfire/sas_gemfire_config.xml]: Cannot resolve reference to bean 'sas_gemfire_cache' while setting bean property 'cache'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sas_gemfire_cache': FactoryBean threw exception on object creation; nested exception is com.gemstone.gemfire.SystemConnectException: Unable to find a free port in the membership-port-range
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:334)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1419)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1160)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:195)
        at org.springframework.context.support.AbstractApplicationContext.registerBeanPostProcessors(AbstractApplicationContext.java:762)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:467)
        at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:410)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
        at com.sas.svcs.webapp.servlet.springframework.DefaultContextLoaderListener.contextInitialized(DefaultContextLoaderListener.java:44)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4792)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5256)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
        at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:629)
        at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1839)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sas_gemfire_cache': FactoryBean threw exception on object creation; nested exception is com.gemstone.gemfire.SystemConnectException: Unable to find a free port in the membership-port-range
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:173)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:102)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1467)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:304)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)
        ... 28 more
Caused by: com.gemstone.gemfire.SystemConnectException: Unable to find a free port in the membership-port-range
        at com.gemstone.gemfire.internal.SocketCreator.createServerSocketUsingPortRange(SocketCreator.java:873)
        at com.gemstone.gemfire.internal.tcp.TCPConduit.createServerSocket(TCPConduit.java:397)
        at com.gemstone.gemfire.internal.tcp.TCPConduit.startAcceptor(TCPConduit.java:340)
        at com.gemstone.gemfire.internal.tcp.TCPConduit.<init>(TCPConduit.java:247)
        at com.gemstone.gemfire.distributed.internal.direct.DirectChannel.<init>(DirectChannel.java:151)
        at com.gemstone.gemfire.distributed.internal.membership.jgroup.JGroupMembershipManager.initialize(JGroupMembershipManager.java:1657)
        at com.gemstone.gemfire.distributed.internal.membership.jgroup.JGroupMemberFactory.newMembershipManager(JGroupMemberFactory.java:112)
        at com.gemstone.gemfire.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:89)
        at com.gemstone.gemfire.distributed.internal.DistributionManager.<init>(DistributionManager.java:1215)
        at com.gemstone.gemfire.distributed.internal.DistributionManager.create(DistributionManager.java:610)
        at com.gemstone.gemfire.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:623)
        at com.gemstone.gemfire.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:282)
        at com.gemstone.gemfire.distributed.DistributedSystem.connect(DistributedSystem.java:1651)
        at com.gemstone.gemfire.cache.CacheFactory.create(CacheFactory.java:227)
        at org.springframework.data.gemfire.CacheFactoryBean.createCache(CacheFactoryBean.java:407)
        at org.springframework.data.gemfire.CacheFactoryBean.init(CacheFactoryBean.java:279)
        at org.springframework.data.gemfire.CacheFactoryBean.getObject(CacheFactoryBean.java:451)
        at org.springframework.data.gemfire.CacheFactoryBean.getObject(CacheFactoryBean.java:76)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:166)
        ... 33 more
2020-09-03 10:58:00,175 ERROR (localhost-startStop-5) [org.apache.catalina.core.StandardContext] One or more listeners failed to start. Full details will be found in the appropriate container log file

In the error text, you can see that the problem may be related to gemfire.
Looking at the gemfire.log, you can see the following error: 

[info 2020/09/11 15:41:43.965 MSK  <localhost-startStop-7> tid=0x1f] Using license-working-dir "/sas/sasconfig/Lev1/Web/WebAppServer/SASServer1_1/logs".

[info 2020/09/11 15:41:43.966 MSK  <localhost-startStop-7> tid=0x1f] Acquiring Application Cache Node license with serial number "6M0C3-4VW9H-M8J40-0D52F-DTM0H".

[info 2020/09/11 15:41:44.064 MSK  <localhost-startStop-7> tid=0x1f] vFabric Licensing Client activation of license required 98 milliseconds.

[info 2020/09/11 15:41:44.076 MSK  <localhost-startStop-7> tid=0x1f] Licensing required 111 milliseconds.

[error 2020/09/11 15:41:45.725 MSK  <localhost-startStop-7> tid=0x1f] Licensing Client: Failed to set component state to 'off': : com.springsource.vfabric.licensing.clien
t.LocalLicenseConfigurationException: Failed to access/create local file /sas/sasconfig/Lev1/Web/WebAppServer/SASServer1_1/logs/vf.gf.acn-license.cfg: java.io.IO
Exception: Too many open files

[error 2020/09/11 15:41:45.725 MSK  <localhost-startStop-7> tid=0x1f] Licensing Client: Failed to get running count from license state: com.springsource.vfabric.licensing
.client.LocalLicenseConfigurationException: Read: Failed to save/load state:java.io.FileNotFoundException: /sas/sasconfig/Lev1/Web/WebAppServer/SASServer1_1/logs
/vf.gf.acn-license.cfg (Too many open files)

[error 2020/09/11 15:41:45.725 MSK  <localhost-startStop-7> tid=0x1f] Licensing Client: Failed to get running count from license state: com.springsource.vfabric.licensing
.client.LocalLicenseConfigurationException: Read: Failed to save/load state:java.io.FileNotFoundException: /sas/sasconfig/Lev1/Web/WebAppServer/SASServer1_1/logs
/vf.gf.acn-license.cfg (Too many open files)

[error 2020/09/11 15:41:45.726 MSK  <localhost-startStop-7> tid=0x1f] Licensing Client: Failed to retrieve evaluation information from saved state: com.springsource.vfabr
ic.licensing.client.LocalLicenseConfigurationException: Read: Failed to save/load state:java.io.FileNotFoundException: /sas/sasconfig/Lev1/Web/WebAppServer/SASSe
rver1_1/logs/vf.gf.acn-license.cfg (Too many open files)

[error 2020/09/11 15:41:45.726 MSK  <localhost-startStop-7> tid=0x1f] Licensing Client: Failed to retrieve evaluation information from saved state: com.springsource.vfabr
ic.licensing.client.LocalLicenseConfigurationException: Read: Failed to save/load state:java.io.FileNotFoundException: /sas/sasconfig/Lev1/Web/WebAppServer/SASSe
rver1_1/logs/vf.gf.acn-license.cfg (Too many open files)

[error 2020/09/11 15:41:45.726 MSK  <localhost-startStop-7> tid=0x1f] Licensing Client: Failed to get running count from license state: com.springsource.vfabric.licensing
.client.LocalLicenseConfigurationException: Read: Failed to save/load state:java.io.FileNotFoundException: /sas/sasconfig/Lev1/Web/WebAppServer/SASServer1_1/logs
/vf.gf.acn-license.cfg (Too many open files)

[error 2020/09/11 15:41:45.726 MSK  <localhost-startStop-7> tid=0x1f] Licensing Client: Failed to retrieve evaluation information from saved state: com.springsource.vfabr
ic.licensing.client.LocalLicenseConfigurationException: Read: Failed to save/load state:java.io.FileNotFoundException: /sas/sasconfig/Lev1/Web/WebAppServer/SASSe
rver1_1/logs/vf.gf.acn-license.cfg (Too many open files)

[error 2020/09/11 15:41:45.726 MSK  <localhost-startStop-7> tid=0x1f] Licensing Client: Failed to apply component change to licenses: : com.springsource.vfabric.licensing
.client.LocalLicenseConfigurationException: Failed to access/create local file /sas/sasconfig/Lev1/Web/WebAppServer/SASServer1_1/logs/vf.gf.acn-license.cfg: java
.io.IOException: Too many open files

[error 2020/09/11 15:41:45.727 MSK  <localhost-startStop-7> tid=0x1f] Licensing Client: Failed to write event to /sas/sasconfig/Lev1/Web/WebAppServer/SASServer1_
1/logs/vf.gf.acn-events.txt: java.io.FileNotFoundException: /sas/sasconfig/Lev1/Web/WebAppServer/SASServer1_1/logs/vf.gf.acn-events.txt (Too many open files)

[info 2020/09/11 15:41:46.834 MSK  <localhost-startStop-5> tid=0x1d]

This error (Too many open files) has not yet been fixed.

Increasing system limits does not help - the SASServer1_1 process reaches the open file limit regardless of the set upper limit.

 

My suspicions were just the same connected with the wrong start of the SASServer1_1. I have no idea how to fix this yet.

AnandVyas
Ammonite | Level 13

While the SDW is open, you can stop SAS services on middle-tier node and change the limits. Start the gem-fire service manually and see if it starts successfully. Then start all the SAS services and hit retry on the wizard.

 

Set ulimits for SAS Web Applications
(On Linux) On the SAS middle tier system, set ulimit settings in /etc/security/limits.conf for the Installer user account that runs the SAS Web Application Server processes.
For example, here are ulimit settings added to /etc/security/limits.conf for the Installer user account, sas:
sas - nofile 150000
sas - nproc  100000
sas - stack  10240
The NPROC (max number of processes) changes do not take effect unless /etc/security/limits.d/90-nproc.conf (Red Hat Enterprise Linux 6) or /etc/security/limits.d/20-nproc.conf (Red Hat Enterprise Linux 7) is also changed.
For example, to allow 100,000 processes on Red Hat Enterprise Linux 6, make the following change to /etc/security/limits.d/90-nproc.conf:
sas  soft  nproc  100000
Start the locator with the instances/ins_41415/gemfire-locator.sh start command.
HULK
Obsidian | Level 7

 I have increased the limits. Then I started gemfire manually. The component does not start, the log contains the following error with the ports:

 

[info 2020/09/11 18:27:52.239 MSK  <main> tid=0x1] Locator stopped

[severe 2020/09/11 18:27:52.239 MSK  <main> tid=0x1] Could not start locator
com.gemstone.gemfire.SystemConnectException: Unable to find a free port in the membership-port-range
        at com.gemstone.gemfire.internal.SocketCreator.createServerSocketUsingPortRange(SocketCreator.java:873)
        at com.gemstone.gemfire.internal.tcp.TCPConduit.createServerSocket(TCPConduit.java:397)
        at com.gemstone.gemfire.internal.tcp.TCPConduit.startAcceptor(TCPConduit.java:340)
        at com.gemstone.gemfire.internal.tcp.TCPConduit.<init>(TCPConduit.java:247)
        at com.gemstone.gemfire.distributed.internal.direct.DirectChannel.<init>(DirectChannel.java:151)
        at com.gemstone.gemfire.distributed.internal.membership.jgroup.JGroupMembershipManager.initialize(JGroupMembershipManager.java:1657)
        at com.gemstone.gemfire.distributed.internal.membership.jgroup.JGroupMemberFactory.newMembershipManager(JGroupMemberFactory.java:112)
        at com.gemstone.gemfire.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:89)
        at com.gemstone.gemfire.distributed.internal.DistributionManager.<init>(DistributionManager.java:1215)
        at com.gemstone.gemfire.distributed.internal.DistributionManager.create(DistributionManager.java:610)
        at com.gemstone.gemfire.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:623)
        at com.gemstone.gemfire.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:282)
        at com.gemstone.gemfire.distributed.DistributedSystem.connect(DistributedSystem.java:1651)
        at com.gemstone.gemfire.distributed.internal.InternalLocator.startDistributedSystem(InternalLocator.java:827)
        at com.gemstone.gemfire.distributed.internal.InternalLocator.startLocator(InternalLocator.java:421)
        at com.gemstone.gemfire.distributed.internal.InternalLocator.startLocator(InternalLocator.java:351)
        at com.gemstone.gemfire.internal.DistributionLocator.main(DistributionLocator.java:148)
        at com.gemstone.gemfire.distributed.Locator.main(Locator.java:544)
AnandVyas
Ammonite | Level 13

Is the locator on compute node up and running? If not, what logs do you see there? Is there any firewall between the machines?

HULK
Obsidian | Level 7

This is an all-in-one installation. Therefore, there is hardly a problem with the presence of any firewalls.

HULK
Obsidian | Level 7

Maybe anyone else has any ideas?

JuanS_OCS
Amethyst | Level 16

Hi @HULK ,

 

yes, your original issue happens because SASServer1_1, where your SAS Content Server is hosted, cannot start. And this happens because the Cache Locator (gemfire), cannot start.

 

When you are troubleshooting issues it is critical to assume nothing. I think it is still worth to check if network it is an issue. It could be a firewall issue, or something else. It is simple, the error below it is self-explanatory:

Unable to find a free port in the membership-port-range

Google it and you will get: Modify the Cache Locator configuration to accommodate a firewall https://documentation.sas.com/?docsetId=bimtag&docsetTarget=n1dvn134xe84k0n1j14o0k534kv5.htm&docsetV...

Pretty useful document.

 

I have a few questions: 

 

  • what is the maintenance level of this as 9.4? SAS 9.4M6? M7?
  • how many IPs/NiC has this server got?
  • Is any of them a public IP address?
  • do you have the server firewalls (firewalld/iptables/other) enabled?
  • on what IP are your web services (and gemfire) listening?

 

You should check as well https://support.sas.com/kb/52/670.html

 

 

 

 

 

 

HULK
Obsidian | Level 7

Hi!

 

Your message gave me an idea. Yes, the server really has two addresses: internal and public.
In the gemfire launch logs I see that the public address of my server is being used:

 

...
Starting peer location for Distribution Locator on my.public.ip.address[41415]
...

I want to try changing it to internal, but have not yet found a way to do it.

Can you tell me in which configuration file i can fix this?

JuanS_OCS
Amethyst | Level 16
Hi!

The link I shared earlier will tell you the files.

You can also grep through your SASCONFIG folder using the port number as string pattern to find all related files 😉
HULK
Obsidian | Level 7

Hi!

 

I have read this document carefully again

 

https://support.sas.com/kb/52/670.html

 

Then forcibly specified the internal IP address of the server in all configuration files. After that, the launch of the component was successful.

 

Thanks for the idea for multiple IP addresses!

JuanS_OCS
Amethyst | Level 16

Great news @HULK ! Are you able to continue the installation process?

HULK
Obsidian | Level 7
Yes, after that the installation process completed without errors.

suga badge.PNGThe SAS Users Group for Administrators (SUGA) is open to all SAS administrators and architects who install, update, manage or maintain a SAS deployment. 

Join SUGA 

Get Started with SAS Information Catalog in SAS Viya

SAS technical trainer Erin Winters shows you how to explore assets, create new data discovery agents, schedule data discovery agents, and much more.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 13 replies
  • 6057 views
  • 4 likes
  • 3 in conversation