Hi Jeff, Thank you very much for suggestions. My focus is now to solve the problem on the physical hadoop cluster. Going into a deep analysis of the problem we found a possible issue in the network communication. In particular, seems to to be necessary a complete visibility between the SAS Client and all HDFS datanode . Could you confirm this ? The SAS client needs to reach all datanode to the HDFS port (50010 / 50020 / 50075) ? There are other service / port on data nodes that need to be reached ? Now we can reach only the name node at port 8020 and hive server at  port 10000 but there is firewall rules between our client and the data nodes of the cluster. Thank you D. ... View more

When SAS sas code is running I found this message several times in the log file of Cloudera Cluster ( non VM 😞 Retry#0: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported in state standby 2015-03-26 11:40:57,582 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hdfs (auth:SIMPLE) cause:org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported in state standby ... View more

Hi Jeff, i'm using  9.4 M2 I have the same problem even if the sticky bit is set to off.( or by changing the HDFS_TEMP path). Opening the log file of the VM i found this message: 2015-03-26 02:21:50,793 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: there are no corrupt file blocks. 2015-03-26 02:21:59,029 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds 2015-03-26 02:21:59,029 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s). 2015-03-26 02:22:29,029 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds 2015-03-26 02:22:29,029 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). 2015-03-26 02:22:50,818 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: there are no corrupt file blocks. 2015-03-26 02:22:51,058 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 97 Total time for transactions(ms): 61 Number of transactions batched in Syncs: 7 Number of syncs: 84 SyncTimes(ms): 289 2015-03-26 02:22:51,472 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /tmp/sasdata-2015-03-26-07-20-34-420-e-00003.dlv. BP-150411824-127.0.0.1-1418915217884 blk_1073742312_1498{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-58c87cab-9390-42e3-ba20-58d96ba4696e:NORMAL|RBW]]} 2015-03-26 02:22:52,376 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to place enough replicas, still in need of 1 to reach 1. For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy 2015-03-26 02:22:52,376 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:cloudera (auth:SIMPLE) cause:java.io.IOException: File /tmp/sasdata-2015-03-26-07-20-34-420-e-00003.dlv could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and 1 node(s) are excluded in this operation. 2015-03-26 02:22:52,376 INFO org.apache.hadoop.ipc.Server: IPC Server handler 16 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 192.168.100.1:49699 Call#30 Retry#0 java.io.IOException: File /tmp/sasdata-2015-03-26-07-20-34-420-e-00003.dlv could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and 1 node(s) are excluded in this operation.   at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1504)   at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3065)   at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:615)   at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:188)   at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:476)   at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)   at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)   at java.security.AccessController.doPrivileged(Native Method)   at javax.security.auth.Subject.doAs(Subject.java:415)   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) 2015-03-26 02:22:52,629 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:cloudera (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException: User does not belong to hive 2015-03-26 02:22:52,630 INFO org.apache.hadoop.ipc.Server: IPC Server handler 16 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.setOwner from 127.0.0.1:41381 Call#191 Retry#0: org.apache.hadoop.security.AccessControlException: User does not belong to hive 2015-03-26 02:22:59,035 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30006 milliseconds 2015-03-26 02:22:59,035 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). 2015-03-26 02:23:29,039 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30004 milliseconds 2015-03-26 02:23:29,039 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s). 2015-03-26 02:23:50,792 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: there are no corrupt file blocks. ... View more

I'm able to connect to the name node and open this logs: [tucloud@usqrtpl5394 hadoop-hdfs]$ pwd /var/log/hadoop-hdfs [tucloud@usqrtpl5394 hadoop-hdfs]$ ls hadoop-cmf-hdfs-DATANODE-usqrtpl5394.internal.unicreditgroup.eu.log.out            hdfs-audit.log hadoop-cmf-hdfs-FAILOVERCONTROLLER-usqrtpl5394.internal.unicreditgroup.eu.log.out  hdfs-audit.log.1 hadoop-cmf-hdfs-JOURNALNODE-usqrtpl5394.internal.unicreditgroup.eu.log.out         hdfs-audit.log.2 hadoop-cmf-hdfs-NAMENODE-usqrtpl5394.internal.unicreditgroup.eu.log.out            SecurityAuth-hdfs.audit hadoop-cmf-hdfs-NAMENODE-usqrtpl5394.internal.unicreditgroup.eu.log.out.1          stacks hadoop-cmf-hdfs-NAMENODE-usqrtpl5394.internal.unicreditgroup.eu.log.out.2 [tucloud@usqrtpl5394 hadoop-hdfs]$ but no error found (tail on log file when the sas script is running) Is /var/log/hadoop-hdfs the logs directory ? ... View more

Hi, thanks for the quick response This is the situation: [tucloud@usqrtpl5394 ~]$ hdfs dfs -ls / Found 5 items drwxrwxrwx   - hdfs  supergroup          0 2015-03-18 18:50 /crmon drwxrwxrwx   - hdfs  supergroup          0 2015-02-16 10:51 /h2oTEMP drwxr-xr-x   - sasep sasep               0 2015-02-27 15:21 /sas drwxrwxrwt   - hdfs  supergroup          0 2015-03-25 15:33 /tmp drwxr-xr-x   - hdfs  supergroup          0 2015-02-10 13:21 /user sticky bit it's on but i have the same problem if i define HDFS_TEMPDIR parameter in the libname statment: libname hive hadoop server='usqrtpl5396.internal.unicreditgroup.eu' user=tucloud HDFS_TEMPDIR="/user/tucloud" database=crmon subprotocol=hive2; I can reach any node with ping cmd from my sas server. Thanks D. ... View more

Hi all, I have the same problem on both virtual machines and on a physical cluster .When i try to load some data in hadoop Cloudera, using a simple datastep, no data is written (Table with zero rows) but the metadata structure of the table is correctly stored (the SAS log is without errors). Do you have suggestions to solve the problem on the hadoop cluster ? Thanks. D. ... View more