rsubmit fails in Dev 9.4 Grid environment; successful in Prod 9.4 Grid environment.

[DanielleCWhite]

I have an open tech support track on this but wanted to ask here as well. One of our data scientists is working on SAS Code which uses rsubmit and has run into an issue where it works in Prod but not Dev. He gave me simplified code that demonstrates the issue and I'm able to repeat it with my credentials. My searches for the errors I'm encountering have not thus far yielded information that assists, e.g. I got a SAS note about a hotfix for 9.4M0.

The environments are SAS 9.4M7 with hotfixes current as of the time of their upgrades (mid-December of last year for Dev and February of this year for Prod.) The grid is SAS Workload Orchestrator rather than LSF. Integrated Windows Authentication (IWA) is used for most client access, particularly via Enterprise Guide. The environments were installed several months apart over a year ago. I was not on this client when either install was done but was for the upgrades. I have observed some configuration differences between environments such as log locations and file system layouts on certain nodes (particularly midtier.) 

The two data files are trivial example data; both are two columns with 3713 rows. The columns are "date" which contains dates in ddMMMyyyy format starting with 01JUL2020. The task column has the same value for every row; 1 in "task1_data" and 2 in "task2_data."

So far with SAS Technical Support the focus has been on the metadata identity having an external account and password provided. By default the account tab has the username but not the password since SAS is configured to use PAM, and the metadata identities are created the same in both environments. As a diagnostic step I added my password to the external credentials in my Dev metadata identity then started a new session in EG without any change in behavior.

The demo code (note: edited to remove my username.)

LIBNAME bee BASE "/sasdata/sasusers/username/BEE";

data work.dates ;
	format date date10.;
	do date=22097 to 22159+(365*10);
		output;
	end;
run;



/*start rsubmit code*/
/*%let rc=%sysfunc(grdsvc_enable(_all_,server=SASApp));*/
%let rc=%sysfunc(grdsvc_enable(_all_, resource=SASApp));

options autosignon;

libname shared "%sysfunc(pathname(work))";



rsubmit task1 wait=no inheritlib=(shared);

LIBNAME bee BASE "/sasdata/sasusers/username/BEE";
data bee.task1_data ;
	set shared.dates;
	task=1;
run;

endrsubmit;



rsubmit task2 wait=no inheritlib=(shared);

LIBNAME bee BASE "/sasdata/sasusers/username/BEE";
data bee.task2_data ;
	set shared.dates;
	task=2;
run;

endrsubmit;

waitfor _all_ task1 task2;
signoff _all_;



data work.shared_combined;
 set bee.task2_data bee.task2_data;
run;

This is the log (with replacements of username and identifying hostnames)

1                                                          The SAS System                         Thursday, May 13, 2021 11:38:00 AM

1          ;*';*";*/;quit;run;
2          OPTIONS PAGENO=MIN;
3          %LET _CLIENTTASKLABEL='rsubmit_example.sas';
4          %LET _CLIENTPROCESSFLOWNAME='Standalone Not In Project';
5          %LET _CLIENTPROJECTPATH='';
6          %LET _CLIENTPROJECTPATHHOST='';
7          %LET _CLIENTPROJECTNAME='';
8          %LET _SASPROGRAMFILE='C:\Users\username\Documents\rsubmit_example.sas';
9          %LET _SASPROGRAMFILEHOST='laptop_24558';
10         
11         ODS _ALL_ CLOSE;
12         OPTIONS DEV=SVG;
13         GOPTIONS XPIXELS=0 YPIXELS=0;
14         %macro HTML5AccessibleGraphSupported;
15             %if %_SAS_VERCOMP_FV(9,4,4, 0,0,0) >= 0 %then ACCESSIBLE_GRAPH;
16         %mend;
17         FILENAME EGHTML TEMP;
18         ODS HTML5(ID=EGHTML) FILE=EGHTML
19             OPTIONS(BITMAP_MODE='INLINE')
20             %HTML5AccessibleGraphSupported
21             ENCODING='utf-8'
22             STYLE=HTMLBlue
23             NOGTITLE
24             NOGFOOTNOTE
25             GPATH=&sasworklocation
26         ;
NOTE: Writing HTML5(EGHTML) Body file: EGHTML
27         
28         LIBNAME bee BASE "/sasdata/sasusers/username/BEE";
NOTE: Libref BEE was successfully assigned as follows: 
      Engine:        BASE 
      Physical Name: /sasdata/sasusers/username/BEE
29         
30         data work.dates ;
31         	format date date10.;
32         	do date=22097 to 22159+(365*10);
33         		output;
34         	end;
35         run;

NOTE: The data set WORK.DATES has 3713 observations and 1 variables.
NOTE: MVA_DSIO.OPEN_CLOSE| _DISARM|         STOP| _DISARM| 2021-05-13T11:38:23,057-05:00| _DISARM| WorkspaceServer| _DISARM| | 
      _DISARM| | _DISARM| | _DISARM| 19820544| _DISARM| 13| _DISARM| 14| _DISARM| 136| _DISARM| 1040| _DISARM| | _DISARM| | 
      _DISARM| | _DISARM| | _DISARM| | _DISARM| | _ENDDISARM 
NOTE: PROCEDURE| _DISARM|         STOP| _DISARM| 2021-05-13T11:38:23,058-05:00| _DISARM| WorkspaceServer| _DISARM| | _DISARM| | 
      _DISARM| | _DISARM| 20082688| _DISARM| 13| _DISARM| 14| _DISARM| 136| _DISARM| 1040| _DISARM| | _DISARM| | _DISARM| | 
      _DISARM| | _DISARM| | _DISARM| | _ENDDISARM 
NOTE: DATA statement used (Total process time):
      real time           0.00 seconds
      cpu time            0.00 seconds
      

36         
37         
38         
39         /*start rsubmit code*/
40         /*%let rc=%sysfunc(grdsvc_enable(_all_,server=SASApp));*/
41         %let rc=%sysfunc(grdsvc_enable(_all_, resource=SASApp));
2                                                          The SAS System                         Thursday, May 13, 2021 11:38:00 AM

42         
43         options autosignon;
44         
45         libname shared "%sysfunc(pathname(work))";
NOTE: Libref SHARED refers to the same physical library as WORK.
NOTE: Libref SHARED was successfully assigned as follows: 
      Engine:        V9 
      Physical Name: 
      /saswork/SAS_work58100000148C_gridhost2.example.com/SAS_workB2EA0000148C_gridhost2.example.com
46         
47         
48         
49         rsubmit task1 wait=no inheritlib=(shared);
NOTE: Remote session ID TASK1 will use the grid service _ALL_.
NOTE: Remote signon to TASK1 commencing (SAS Release 9.04.01M7P080520).
ERROR: PROVIDER(SAS): Unable to get grid information from the master host.
ERROR: Cannot initialize the grid provider module.
ERROR: A communication subsystem partner link setup request failure has occurred.
ERROR: Cannot initialize the grid provider module.
NOTE: SAS.Connect.SIGNON| _DISARM|         STOP| _DISARM| 2021-05-13T11:38:23,203-05:00| _DISARM| SAS/CONNECT:TASK1| _DISARM| SAS| 
      _DISARM| | _DISARM| | _DISARM| | _DISARM| | _DISARM| | _DISARM| | _DISARM| | _DISARM| 0.010000| _DISARM| 0.040638| _DISARM| 
      1936543103.162876| _DISARM| 1936543103.203514| _DISARM| 0.010000| _DISARM| | _ENDDISARM 
NOTE: Subsequent lines will be ignored until ENDRSUBMIT.
ERROR: Remote submit to TASK1 canceled.
50         
51         
52         
53         rsubmit task2 wait=no inheritlib=(shared);
NOTE: Remote session ID TASK2 will use the grid service _ALL_.
NOTE: Remote signon to TASK2 commencing (SAS Release 9.04.01M7P080520).
ERROR: PROVIDER(SAS): Unable to get grid information from the master host.
ERROR: Cannot initialize the grid provider module.
ERROR: A communication subsystem partner link setup request failure has occurred.
ERROR: Cannot initialize the grid provider module.
NOTE: SAS.Connect.SIGNON| _DISARM|         STOP| _DISARM| 2021-05-13T11:38:23,233-05:00| _DISARM| SAS/CONNECT:TASK2| _DISARM| SAS| 
      _DISARM| | _DISARM| | _DISARM| | _DISARM| | _DISARM| | _DISARM| | _DISARM| | _DISARM| 0.000000| _DISARM| 0.023485| _DISARM| 
      1936543103.210368| _DISARM| 1936543103.233853| _DISARM| 0.000000| _DISARM| | _ENDDISARM 
NOTE: Subsequent lines will be ignored until ENDRSUBMIT.
ERROR: Remote submit to TASK2 canceled.
NOTE: There are no active tasks/transactions.
54         
55         waitfor _all_ task1 task2;
56         signoff _all_;
NOTE: No active remote sessions exist; SIGNOFF is not necessary.
57         
58         
59         
60         data work.shared_combined;
61          set bee.task2_data bee.task2_data;
62         run;

NOTE: There were 3713 observations read from the data set BEE.TASK2_DATA.
NOTE: MVA_DSIO.OPEN_CLOSE| _DISARM|         STOP| _DISARM| 2021-05-13T11:38:23,250-05:00| _DISARM| WorkspaceServer| _DISARM| | 
      _DISARM| | _DISARM| | _DISARM| 20344832| _DISARM| 13| _DISARM| 17| _DISARM| 136| _DISARM| 1176| _DISARM| | _DISARM| | 
      _DISARM| | _DISARM| | _DISARM| | _DISARM| | _ENDDISARM 
NOTE: There were 3713 observations read from the data set BEE.TASK2_DATA.
NOTE: MVA_DSIO.OPEN_CLOSE| _DISARM|         STOP| _DISARM| 2021-05-13T11:38:23,250-05:00| _DISARM| WorkspaceServer| _DISARM| | 
      _DISARM| | _DISARM| | _DISARM| 20344832| _DISARM| 13| _DISARM| 17| _DISARM| 136| _DISARM| 1176| _DISARM| | _DISARM| | 
      _DISARM| | _DISARM| | _DISARM| | _DISARM| | _ENDDISARM 
NOTE: The data set WORK.SHARED_COMBINED has 7426 observations and 2 variables.
NOTE: MVA_DSIO.OPEN_CLOSE| _DISARM|         STOP| _DISARM| 2021-05-13T11:38:23,251-05:00| _DISARM| WorkspaceServer| _DISARM| | 
      _DISARM| | _DISARM| | _DISARM| 20344832| _DISARM| 13| _DISARM| 17| _DISARM| 264| _DISARM| 1304| _DISARM| | _DISARM| | 
      _DISARM| | _DISARM| | _DISARM| | _DISARM| | _ENDDISARM 
NOTE: PROCEDURE| _DISARM|         STOP| _DISARM| 2021-05-13T11:38:23,251-05:00| _DISARM| WorkspaceServer| _DISARM| | _DISARM| | 
3                                                          The SAS System                         Thursday, May 13, 2021 11:38:00 AM

      _DISARM| | _DISARM| 20344832| _DISARM| 13| _DISARM| 17| _DISARM| 264| _DISARM| 1304| _DISARM| | _DISARM| | _DISARM| | 
      _DISARM| | _DISARM| | _DISARM| | _ENDDISARM 
NOTE: DATA statement used (Total process time):
      real time           0.01 seconds
      cpu time            0.01 seconds
      

63         
64         %LET _CLIENTTASKLABEL=;
65         %LET _CLIENTPROCESSFLOWNAME=;
66         %LET _CLIENTPROJECTPATH=;
67         %LET _CLIENTPROJECTPATHHOST=;
68         %LET _CLIENTPROJECTNAME=;
69         %LET _SASPROGRAMFILE=;
70         %LET _SASPROGRAMFILEHOST=;
71         
72         ;*';*";*/;quit;run;
73         ODS _ALL_ CLOSE;
74         
75         
76         QUIT; RUN;
77         

[CarmineVerrell]

Not sure what the problem is.

Here are a coupe of questions;

Can the Data Scientist execute anything on Dev(using Rsubmit)?

Can anyone submit on Dev in the same manner?

[DanielleCWhite]

I believe this is the first anyone tried to use rsubmit in either environment. I attempted running code that has a single rsubmit task and just runs proc setinit which also failed.

Due in part to a happy coincidence of having trace logging enabled on the Workspace Server for another TS track I just resolved the problem. I'll put the solution in a separate reply.

[DanielleCWhite · Accepted solution]

I just managed to resolve this. I had enabled trace logging on the SASApp Workspace Server as requested by SAS Technical Support in a track on an unrelated issue and looked at the log produced from running this which contained the following.

NOTE: Remote session ID TASK1 will use the grid service _ALL_.
NOTE: Remote signon to TASK1 commencing (SAS Release 9.04.01M7P080520).
Secure communications error status 807ff01d description "[ip address redacted]: SSL Error: Cannot use private key file; please check your password."
SSL Error: Cannot use private key file; please check your password.
ERROR: Secure communications error status 807ff01d description "[ip address redacted]: SSL Error: Cannot use private key file; please check your password."
ERROR: SSL Error: Cannot use private key file; please check your password.
ERROR: PROVIDER(SAS): Unable to get grid information from the master host.
ERROR: Cannot initialize the grid provider module.
ERROR: A communication subsystem partner link setup request failure has occurred.
ERROR: Cannot initialize the grid provider module.
NOTE: Subsequent lines will be ignored until ENDRSUBMIT.
ERROR: Remote submit to TASK1 canceled.

After a bit of research I found the problem: the SSL key file for the SSL certificate used by SAS Workload Orchestrator had 700 permissions (rwx------.) In Prod it was 644 (rw-r--r--.) I changed it to 644 in Dev and rsubmit now works.