If I submit a program in batch mode that takes a week to finish and, during that week I submit another program in batch mode, will it immediately execute that program or will it wait until the 1st one finishes? If the former, how does running two progs in parallel impact the computer's resources. In particular, a 16-core CPU that is only using 4% of its capacity on the first program. If the programs are I/O intensive, would they compete? Would adding a second drive and directing the output of the second program to the second drive reduce the I/O competition? Would I be worse off running both in parallel rather than in series? Thanks
If I submit a program in batch mode that takes a week to finish and, during that week I submit another program in batch mode, will it immediately execute that program or will it wait until the 1st one finishes?
It will immediately execute as a separate process.
how does running two progs in parallel impact the computer's resources
Depends. On your 16-core CPU the first program is probably only using one or two CPUs, so starting another batch program should not affect it negatively.
If the programs are I/O intensive, would they compete?
They may. In which case you should look at the disk usage under the "Performance" tab in the task manager, to see what disks are heavily used, and under "Processes" to see which programs are disk heavy.
Would adding a second drive and directing the output of the second program to the second drive reduce the I/O competition?
It might, depends on how hard the I/O competition is.
Would I be worse off running both in parallel rather than in series?
I my experience, rarely. Competition for disk and CPU might make both jobs run slower than if they had the computer to themselves, but total elapsed times for the two jobs will probably be lower. If you are short on memory there may be a problem, though, as starting too many processes will cause the computer to swap more working memory to and from disks, which can bring the machine to an almost complete standstill.
If I submit a program in batch mode that takes a week to finish and, during that week I submit another program in batch mode, will it immediately execute that program or will it wait until the 1st one finishes?
It will immediately execute as a separate process.
how does running two progs in parallel impact the computer's resources
Depends. On your 16-core CPU the first program is probably only using one or two CPUs, so starting another batch program should not affect it negatively.
If the programs are I/O intensive, would they compete?
They may. In which case you should look at the disk usage under the "Performance" tab in the task manager, to see what disks are heavily used, and under "Processes" to see which programs are disk heavy.
Would adding a second drive and directing the output of the second program to the second drive reduce the I/O competition?
It might, depends on how hard the I/O competition is.
Would I be worse off running both in parallel rather than in series?
I my experience, rarely. Competition for disk and CPU might make both jobs run slower than if they had the computer to themselves, but total elapsed times for the two jobs will probably be lower. If you are short on memory there may be a problem, though, as starting too many processes will cause the computer to swap more working memory to and from disks, which can bring the machine to an almost complete standstill.
Yes, some definite degradation in run time when running multiple batches, but that is more than made up for in total overall time reduction compared to running in series.
CPU usage was less than expected. For example, one prog used 3-4% of CPU. However, running 26 progs in parallel only used 50-60% of CPU. (Although I did notice that CORE 14 was bumping along at 90-100%; slightly less pure silicone I guess.)
However, with 26 progs running, initial sampling showed the run time was only double that of running a single instance. Suggesting lots of I/O headroom. Thanks
You should still take a look at waitstates, I think that you should get a higher CPU consumption form a process.
We run a 2-core POWER8 AIX, and a single SAS process routinely gets over 50% CPU (more than one core) with a rather moderate I/O throughput.
Your CPU load of less than a full core lets me suspect lots of wait states.
I use topas (out-of-the-box system utility) on my AIX server to display important server performance information, the display looks like this:
Topas Monitor for host:xxxxxxxx EVENTS/QUEUES FILE/TTY Thu Dec 19 07:27:34 2019 Interval:2 Cswitch 1012 Readch 4234 Syscall 4097 Writech 11039 Kernel 2.54 |# | Reads 17 Rawin 0 User 1.84 |# | Writes 752 Ttyout 354 Wait 0.28 |# | Forks 0 Igets 0 Idle 95.34 |########################### | Execs 0 Namei 173 Physc = 0.08 %Entc = 8.42 Runqueue 0 Dirblk 0 Waitqueue 0.0 Network BPS I-Pkts O-Pkts B-In B-Out MEMORY Total 176K 187.5 189.0 86.4K 89.7K PAGING Real,MB 81920 Faults 0 % Comp 25 Disk Busy% BPS TPS B-Read B-Writ Steals 0 % Noncomp 13 hdisk101 2.5 60.0K 15.00 0 60.0K PgspIn 0 % Client 13 hdisk104 2.5 60.0K 15.00 0 60.0K PgspOut 0 hdisk1 0.0 0 0 0 0 PageIn 0 PAGING SPACE hdisk0 0.0 0 0 0 0 PageOut 15 Size,MB 98688 hdisk100 0.0 0 0 0 0 Sios 15 % Used 0 hdisk106 0.0 0 0 0 0 % Free 100 hdisk103 0.0 0 0 0 0 AME hdisk105 0.0 0 0 0 0 TMEM 32.00GWPAR Activ 0 cd0 0.0 0 0 0 0 CMEM 16.00MWPAR Total 0 EF[T/A] 2.5/2.5Press: "h"-help FileSystem BPS TPS B-Read B-Writ CI: 0CO: 0 "q"-quit Total 5.54K 742.0 1.20K 4.34K Name PID CPU% PgSp Owner reaffin 524308 2.0 100K root java 48234514 1.3 1.12G sas topas_nm 8192046 0.5 7.44M root java 27918426 0.5 212M sas nfssync_ 2293778 0.3 108K root java 14876674 0.3 209M sas topas 19529824 0.2 5.27M root dtterm 26148896 0.2 1.58M root
You can see the line for the waitstates; at the moment I took this snapshot, the server was quite idle.
There are also the typical issues to consider (not part of your original question).
Is one program tying up a data set that the other program requires? (Sorting a data set usually ties it up.)
Does an output of one program become an input to another program?
@texasmfp wrote:
In particular, a 16-core CPU that is only using 4% of its capacity on the first program.
Which means you already have a problem. 4% is about half a core, very little in light of many SAS procedures being multi-threaded. This points to a lot of I/O waits or delays caused by network transfers (are you reading from a remote DBMS?).
You need to inspect your jobs in this regard.
Build your skills. Make connections. Enjoy creative freedom. Maybe change the world. Registration is now open through August 30th. Visit the SAS Hackathon homepage.
Register today!Learn how use the CAT functions in SAS to join values from multiple variables into a single value.
Find more tutorials on the SAS Users YouTube channel.
Select SAS Training centers are offering in-person courses. View upcoming courses for: