Not enough resources: Simulation and the calculation of pair-wise distances

[Patrick]

Good. Let me know what the result was once your done.

[tesu]

Patrick,

I want to suppress producing unnecessary log files. I erase the following lines from your original code:

options fullstimer msglevel=i;

%let DTstamp=%sysfunc(datetime(), B8601DT.);
%let cmd=&cmd -log "&dir_path\%scan(&pgrm,1,.)_&DTstamp..log
%let cmd=&cmd -print "&dir_path\%scan(&pgrm,1,.)_&DTstamp..lst

status    =S012345_&DTstamp
mname     =taskname

And I added

"options nonotes nosource nosource2 errors=0;"/

after put in hoping not to produce log files for each individual iteration.

But I still see prog1.log, prog2.log, ..., prog1000.log log files in the folder that has the main SAS code. There could be a procedure to direct those log files to other directory. However, I don't need all the log files.

[Patrick]

A SAS job produces a log file. You can direct it to another path and you can also influence the amount of log messages, eg. by using NONOTES (you need to add this to the generated code inside the PUT statement or you pass it in as a parameter to SYSTASK command in the form "-nonotes ".

%let cmd=&cmd -log "&dir_path\logs\%scan(&pgrm,1,.)_&DTstamp..log -nonotes -nosource -nosource2 -errors=0

%let cmd=&cmd -print "&dir_path\output\%scan(&pgrm,1,.)_&DTstamp..lst

Make sure the 2 sub-directories LOGS and OUTPUT exist.

I personlly believe that LOG is valuable information so I wouldn't reduce it too much - and it won't take up a lot of time for SAS to write these logs.

[tesu]

Patrick. The processing time was:

PROCESSING TIME:    107 (mm:ss)

It is a great improvement, but worth trying one more thing. Write a C code to calculate pair-wise distances because the distances calculation is the most time-consuming task. I can write it in C, but I don't know how to use the C in SAS. After I write the C code, I will revisit the SAS community and launch a separate question.

Thank you very much for your help. I learned a lot from you. Good old times...

William

[Patrick]

I'm out of my depth when it comes to C programming. The following link might give you guidance when it comes to calling a DLL from SAS  SAS(R) 9.3 Companion for Windows

You're definitely seeking the challenge here. As much as I know SAS Procedures are using a C compiler. But of course: If you write your code directly in C and for exactly your case then you might very well be able to beat the SAS Proc.

If you succeed: Very curious to learn how much you could improve performance. Please add a link (reference) here in this thread once you post your follow-up question/results so that I don't miss it.