@Tom @PaigeMiller Hi Tom and Paige, thanks for all the comments. It took me a while to reply because I was suspecting the SAS somehow enter a freeze state since my code is only submitted but not run. So I restarted SAS and it took me a while to rerun the previous temp data sets. Then my remote computer had some problems that were fixed until now.  I add the debug option at the beginning and rerun the nested loop, turns out it works as expected! But the debug line is very helpful for me. ... View more

the code first enumerate the number of records each user_id has, then get the max_num of the records, say the user with the most number of records has 79 records, this max_num will be 79. Then there is a nest loop that go from 1 to 79, and split the data into 79 parts. Then I merge these part horizontally together. I tried the code in charge of these functions on one dataset, and it worked as I wanted. So my problem is just the format of nested loop. ... View more

I don't have this line, but before it works fine. What is the function of this line? without it, the program may still run but not write log? Can you point to me which windows event I should look for? Thanks a lot! ... View more

1 The SAS System 18:36 Tuesday, April 9, 2024 NOTE: Unable to open SASUSER.REGSTRY. WORK.REGSTRY will be opened instead. NOTE: All registry changes will be lost at the end of the session. WARNING: Unable to copy SASUSER registry to WORK registry. Because of this, you will not see registry customizations during this session. NOTE: Unable to open SASUSER.PROFILE. WORK.PROFILE will be opened instead. NOTE: All profile changes will be lost at the end of the session. NOTE: Copyright (c) 2016 by SAS Institute Inc., Cary, NC, USA. NOTE: SAS (r) Proprietary Software 9.4 (TS1M6) Licensed to UNIVERSITY OF MARYLAND - T&R SFA, Site 70210321. NOTE: This session is executing on the X64_SRV19 platform. NOTE: Analytical products: SAS/STAT 15.1 SAS/ETS 15.1 SAS/IML 15.1 SAS/QC 15.1 NOTE: Additional host information: X64_SRV19 WIN 10.0.20348 Server NOTE: SAS initialization used: real time 0.46 seconds cpu time 0.37 seconds 1 /* this file takes out those who graduated from their highest degree in 2000*/ 2 libname revelio "C:\Users\cmlei\Box\box-group-smith-finance-resesarch\revelio_data"; NOTE: Libref REVELIO was successfully assigned as follows: Engine: V9 Physical Name: C:\Users\cmlei\Box\box-group-smith-finance-resesarch\revelio_data 3 libname hdrg2000 "D:\ralymax_cm\data\cleaned\hdg_2000_panel"; NOTE: Libref HDRG2000 was successfully assigned as follows: Engine: V9 Physical Name: D:\ralymax_cm\data\cleaned\hdg_2000_panel 4 libname temp "D:\ralymax_cm\temp\hdg_2000_panel"; NOTE: Libref TEMP was successfully assigned as follows: Engine: V9 Physical Name: D:\ralymax_cm\temp\hdg_2000_panel 5 libname linkedin "D:\ralymax_cm\data\raw\inidvidual_position_raw"; NOTE: Libref LINKEDIN was successfully assigned as follows: Engine: V9 Physical Name: D:\ralymax_cm\data\raw\inidvidual_position_raw 6 7 * change to the fast drive; 8 data _null_; 9 rc=dlgcdir("C:\ralymax"); 10 put rc=; 11 run; 2 The SAS System 18:36 Tuesday, April 9, 2024 NOTE: The current working directory is now "C:\ralymax". rc=0 NOTE: DATA statement used (Total process time): real time 0.00 seconds cpu time 0.01 seconds 12 13 /************** ma group **************/ 14 * view vars we have; 15 proc contents data = hdrg2000.edu_uid_hgrd_2000_ma; 16 run; NOTE: PROCEDURE CONTENTS used (Total process time): real time 0.13 seconds cpu time 0.01 seconds NOTE: The PROCEDURE CONTENTS printed page 1. 17 18 * take out their complete career history; 19 data temp.user_ma; 20 set hdrg2000.edu_uid_hgrd_2000_ma (keep = user_id); 21 run; NOTE: There were 229103 observations read from the data set HDRG2000.EDU_UID_HGRD_2000_MA. NOTE: The data set TEMP.USER_MA has 229103 observations and 1 variables. NOTE: DATA statement used (Total process time): real time 0.41 seconds cpu time 0.01 seconds 22 23 proc sort data = temp.user_ma nodupkey dupout = dups; 24 by user_id; 25 run; NOTE: There were 229103 observations read from the data set TEMP.USER_MA. NOTE: 1932 observations with duplicate key values were deleted. NOTE: The data set WORK.DUPS has 1932 observations and 1 variables. NOTE: The data set TEMP.USER_MA has 227171 observations and 1 variables. NOTE: PROCEDURE SORT used (Total process time): real time 0.04 seconds cpu time 0.06 seconds This is what shown in log, no error printed, just stop here and not going forward. I'm running on a server through remote connection from my desktop. Things work fine if I run directly in sas interactive mode, but stuck when i submit multiple codes using batch sequentially ... View more

Hi, there is sas log! But it does not print any error, it just stop in the middle. So I went to check the code in the interactive mode, but everything works fine. ... View more

Hi Ahmed,   Thanks for the thoughts, I don't know the thing about memory to each task thing before. I guess I submit 99 task with 99 big datasets, and I do not have enough memory, so they start to wait in line. Thanks! ... View more

I get rid of the comma and run again, but it still says: ERROR: A communication subsystem partner link setup request failure has occurred. ERROR: Cannot locate TCP host 'TASK10'. ERROR: Remote signon to TASK10 canceled. ERROR: A link must be established by executing the SIGNON command before you can communicate with TASK10. ... View more

HI Ahmed,   can you tell me which part i should replace? and what should I replace it with? I tried the following: signon task10 inheritlib=(raw, clean); %syslput g_pWorkPath = %sysfunc(PATHNAME(WORK)) / remote=task10; rsubmit task10 wait=no; LIBNAME pWork "&g_pWorkPath"; but it tells me: ERROR: Library , has not been assigned in the local session therefore it cannot be inherited by the remote session. ERROR: Remote signon to TASK10 canceled. ERROR: A link must be established by executing the SIGNON command before you can communicate with TASK10. 114 %syslput g_pWorkPath = %sysfunc(PATHNAME(WORK)) / remote=task10; 115 rsubmit task10 wait=no; ERROR: A link must be established by executing the SIGNON command before you can communicate with TASK10. ERROR: Remote submit to TASK10 canceled. ... View more

Hi Tom,   I did tried this way, but it can only inherit non-temporary folder I built myself, the temporary folder cannot be inherited. The detail is: I set the following folder as my work library (the temporary one): E:\data\work and using your code i did this:  libname work 'E:\data\work'; options sascmd="sas"; options CPUCOUNT= 45; /* Current datetime */ %let _start_dt = %sysfunc(datetime()); /* prosess 10 */ signon task10 inheritlib=(work); rsubmit task10 wait=no; .... but SAS told me: ERROR: The SAS system library WORK may not be reassigned. ERROR: Error in the LIBNAME statement.   The above code works for all other non-temporary folder. Just to be clear: by temporary folder/work folder here, i mean the one where the data is stored if you don't specify libname in libname.dataname and all files in this folder will be deleted after every sas session ... View more

These are really helpful (and for me easy to understand check)! Thanks! ... View more

How can I check if it is IO bound or CPU bound? currently i add threads to proc sql, and set options cpucounts = 40, I do find the procedure is much faster. However, in resources manager, it says threads=40, cpu = 1, so im only using one CPU, but my CPU usage is just 3%. Maybe because my code does not need so much CPU so they did not allocate them in multiple cpu? Im not sure how to see if it is IO bound. Can you point to me where?   ... View more