Viya - SAS Studio - Error Loading Data from CAS to Hadoop

[SwapnaSingh]

Hi All,

I have written a code in Viya SAS Studio to load data from CAS to Hadoop. the Issue I am facing with my code is that if I load data below 50 GB it works completely fine but now when I have requirement to load a data of more than 200GB it throws me below error:

ERROR: Communication failure between HDFS and Cloud Analytic Services.

My question:

Have anyone seen this error?

I know I need to set some data limit to make it work, but I tried below options in libname but it did not work:

/* Extend casdatalimit */

options casdatalimit=ALL;

/* Assign libname for public and puhdfs library */

libname public cas caslib=public datalimit=ALL;

libname pubhdfs cas caslib=pubhdfs datalimit=ALL;

proc casutil;
save casdata="xxx" incaslib="public" outcaslib="pubhdfs"
casout="xxx";
quit;

proc casutil;
load casdata="xxx" incaslib="pubhdfs"
outcaslib="pubhdfs" casout="xxx" promote;
run;

Any input and help will be really appreciated.

Thanks in Advance.

Regards,

Swapna

[alexal]

Reproduce the problem again and show me the most recent CAS server logs from the controller as well as from the worker nodes. You can find the CAS server logs in /opt/sas/viya/config/var/log/cas/default directory.

[SwapnaSingh]

Thanks for your response, the log file is too big so sending you the error from the CAS log from server from controller node:

2020-07-24T12:16:54,802 INFO [00000007] 48939 sas.searchIndex 4843 [tkcasaimp.c:4448] - ++ action table.save / table={name='SearchIndex_1184252209598578986'
, caslib='search'}, name='SearchIndex_1184252209598578986', copies=2, replace=true, caslib='search';
2020-07-24T12:16:55,086 INFO [00000007] 48939 sas.searchIndex 4843 [tkcastabi.c:5639] - Cloud Analytic Services saved the file SearchIndex_11842522095985789
86.sashdat in caslib search.
2020-07-24T12:16:55,086 INFO [00000007] 48939 sas.searchIndex 4843 [tkcastabi.c:5643] - Global table SEARCHINDEX_1184252209598578986 in caslib search saved
to /opt/sas/data/search/SearchIndex_1184252209598578986.sashdat.
2020-07-24T12:16:55,094 INFO [00000007] 48939 sas.searchIndex 4843 [tkcasaimp.c:6314] - Network traffic used 4MB sent, 0MB broadcast, 0MB of data reduced.
2020-07-24T12:16:55,094 INFO [00000007] 48939 sas.searchIndex 4843 [tkcasaimp.c:6404] - -- 'table.save' SUCCESSFUL.
2020-07-24T12:16:55,181 INFO [00000007] 48939 sas.searchIndex 4843 [tkcastaba.c:3327] - Accessed session table SEARCHINDEX_1184252209598578986-ATTR loaded in caslib search.
2020-07-24T12:16:55,195 INFO [00000007] 48939 sas.searchIndex 4843 [tkcasaimp.c:4448] - ++ action table.save / table={name='SearchIndex_1184252209598578986-attr', caslib='search'}, name='SearchIndex_1184252209598578986-attr', copies=2, replace=true, caslib='search';
2020-07-24T12:16:55,374 INFO [00000007] 48939 sas.searchIndex 4843 [tkcastabi.c:5639] - Cloud Analytic Services saved the file SearchIndex_1184252209598578986-attr.sashdat in caslib search.
2020-07-24T12:16:55,374 INFO [00000007] 48939 sas.searchIndex 4843 [tkcastabi.c:5646] - Session table SEARCHINDEX_1184252209598578986-ATTR in caslib search saved to /opt/sas/data/search/SearchIndex_1184252209598578986-attr.sashdat.
2020-07-24T12:16:55,380 INFO [00000007] 48939 sas.searchIndex 4843 [tkcasaimp.c:6404] - -- 'table.save' SUCCESSFUL.
2020-07-24T12:16:55,455 INFO [00000007] 48939 sas.searchIndex 4843 [tkcasaimp.c:4448] - ++ action session.endSession;
2020-07-24T12:16:55,469 INFO [00000007] 48939 sas.searchIndex 4843 [tkcasaimp.c:6404] - -- 'session.endSession' SUCCESSFUL.
2020-07-24T12:16:55,471 INFO [00000007] 48939 sas.searchIndex 4843 [tkcsesinst.c:835] - Destroying session 756b3ad3-373f-f441-bde6-993f25f84d7c.
2020-07-24T12:16:55,719 INFO [00072181] MAIN sas.searchIndex 4843 [tkcsesinst.c:835] - Destroying session 756b3ad3-373f-f441-bde6-993f25f84d7c.
2020-07-24T12:18:14,983 ERROR [00000007] 43786 xxx 4838 [tkcastaba.c:28191] - Communication failure between HDFS and Cloud Analytic Services.
2020-07-24T12:18:14,996 INFO [00000007] 43786 xxx 4838 [tkcasaimp.c:6404] - -- 'table.save' FAILED. Communication failure between HDFS and Cloud Analytic Services.
2020-07-24T12:18:15,049 INFO [00000007] 43786 xxx 4838 [tkcasaimp.c:4448] - ++ action table.loadTable / path='xxx.sashdat', caslib='pubhdfs', casOut={name='xxx', caslib='pubhdfs', promote=true};
2020-07-24T12:18:18,829 ERROR [00000007] 43786 xxx 4838 [tkcastaba.c:28110] - The file or path 'xxx.sashdat' is not available in the file system.
2020-07-24T12:18:18,842 INFO [00000007] 43786 xxx 4838 [tkcasaimp.c:6404] - -- 'table.loadTable' FAILED. 0x887ff81b
2020-07-24T12:18:34,169 INFO [00072216] MAIN NoUser MAIN [tkcalsident.c:424] - New client connection accepted on port 5570. Client IP address and port are [xxx]:xxx. TLS is required.
2020-07-24T12:18:34,169 INFO [00072216] MAIN NoUser MAIN [tkident.c:1323] - User xxx successfully authenticated using the OAuth authentication provider.
2020-07-24T12:18:34,169 INFO [00072216] MAIN NoUser MAIN [tkcsesinst.c:748] - Successfully created session e656e96b-69bd-e145-8f49-247067a7e1a5.
2020-07-24T12:18:34,194 INFO [00072216] MAIN xxx 4844 [casgeneral.c:4824] - Launched session controller. Process ID is 55291.
2020-07-24T12:18:34,227 INFO [00000007] 55291 cas MAIN [cas.c:2310] - Starting Cloud Analytic Services driver, host=xxx.ic.ing.net, id=0, role=session controller.
2020-07-24T12:18:34,229 INFO [00000007] 55291 cas MAIN [tkcsesinst.c:748] - Successfully created session e656e96b-69bd-e145-8f49-247067a7e1a5.
2020-07-24T12:18:34,231 INFO [00072217] MAIN NoUser MAIN [tkcalsident.c:424] - New client connection accepted on port 5570. Client IP address and port are [10.36.20.36]:40286. TLS is required.
2020-07-24T12:18:34,231 INFO [00072217] MAIN NoUser MAIN [tkident.c:1323] - User xxx successfully authenticated using the OAuth authentication provider.
2020-07-24T12:18:34,232 INFO [00072217] MAIN NoUser MAIN [tkcsesinst.c:748] - Successfully created session 682d4059-274b-434a-bb1b-1b2a54410112.
2020-07-24T12:18:34,233 INFO [00000007] 55291 xxx 4844 [tkhttpserver.c:1353] - Starting HTTP server 'xxx.ic.ing.net', minPort=0 maxPort=0
:

I checked there is no error in worker nodes.