Running multiple programs in batch mode and computer resources

[texasmfp]

If I submit a program in batch mode that takes a week to finish and, during that week I submit another program in batch mode, will it immediately execute that program or will it wait until the 1st one finishes?  If the former, how does running two progs in parallel impact the computer's resources.  In particular, a 16-core CPU that is only using 4% of its capacity on the first program.  If the programs are I/O intensive, would they compete?  Would adding a second drive and directing the output of the second program to the second drive reduce the I/O competition?  Would I be worse off running both in parallel rather than in series?  Thanks

[s_lassen · Accepted solution]

If I submit a program in batch mode that takes a week to finish and, during that week I submit another program in batch mode, will it immediately execute that program or will it wait until the 1st one finishes? 

It will immediately execute as a separate process.

how does running two progs in parallel impact the computer's resources

Depends. On your 16-core CPU the first program is probably only using one or two CPUs, so starting another batch program should not affect it negatively.

If the programs are I/O intensive, would they compete?

They may. In which case you should look at the disk usage under the "Performance" tab in the task manager, to see what disks are heavily used, and under "Processes" to see which programs are disk heavy.

Would adding a second drive and directing the output of the second program to the second drive reduce the I/O competition?

It might, depends on how hard the I/O competition is.

Would I be worse off running both in parallel rather than in series?

I my experience, rarely. Competition for disk and CPU might make both jobs run slower than if they had the computer to themselves, but total elapsed times for the two jobs will probably be lower. If you are short on memory there may be a problem, though, as starting too many processes will cause the computer to swap more working memory to and from disks, which can bring the machine to an almost complete standstill.

[texasmfp]

Yes, some definite degradation in run time when running multiple batches, but that is more than made up for in total overall time reduction compared to running in series.

CPU usage was less than expected.  For example, one prog used 3-4% of CPU.  However, running 26 progs in parallel only used 50-60% of CPU.  (Although I did notice that CORE 14 was bumping along at 90-100%; slightly less pure silicone I guess.)

However, with 26 progs running, initial sampling showed the run time was only double that of running a single instance.  Suggesting lots of I/O headroom.  Thanks

[Kurt_Bremser]

You should still take a look at waitstates, I think that you should get a higher CPU consumption form a process.

We run a 2-core POWER8 AIX, and a single SAS process routinely gets over 50% CPU (more than one core) with a rather moderate I/O throughput.

Your CPU load of less than a full core lets me suspect lots of wait states.

[texasmfp]

Thanks Kurt. How would I go about looking at "waitstates"?

[Kurt_Bremser]

I use topas (out-of-the-box system utility) on my AIX server to display important server performance information, the display looks like this:

Topas Monitor for host:xxxxxxxx                 EVENTS/QUEUES    FILE/TTY       
Thu Dec 19 07:27:34 2019   Interval:2           Cswitch    1012  Readch     4234
                                                Syscall    4097  Writech   11039
Kernel    2.54  |#                           |  Reads        17  Rawin         0
User      1.84  |#                           |  Writes      752  Ttyout      354
Wait      0.28  |#                           |  Forks         0  Igets         0
Idle     95.34  |########################### |  Execs         0  Namei       173
Physc =   0.08                  %Entc =   8.42  Runqueue      0  Dirblk        0
                                                Waitqueue   0.0                 
Network    BPS  I-Pkts  O-Pkts    B-In   B-Out                   MEMORY         
Total     176K   187.5   189.0   86.4K   89.7K  PAGING           Real,MB   81920
                                                Faults        0  % Comp     25  
Disk    Busy%      BPS     TPS  B-Read  B-Writ  Steals        0  % Noncomp  13  
hdisk101  2.5    60.0K   15.00       0   60.0K  PgspIn        0  % Client   13  
hdisk104  2.5    60.0K   15.00       0   60.0K  PgspOut       0                 
hdisk1    0.0        0       0       0       0  PageIn        0  PAGING SPACE   
hdisk0    0.0        0       0       0       0  PageOut      15  Size,MB   98688
hdisk100  0.0        0       0       0       0  Sios         15  % Used      0  
hdisk106  0.0        0       0       0       0                   % Free    100  
hdisk103  0.0        0       0       0       0  AME             
hdisk105  0.0        0       0       0       0  TMEM       32.00GWPAR Activ    0
cd0       0.0        0       0       0       0  CMEM       16.00MWPAR Total    0
                                                EF[T/A]   2.5/2.5Press: "h"-help
FileSystem          BPS    TPS  B-Read  B-Writ  CI:    0CO:    0        "q"-quit
Total             5.54K  742.0   1.20K   4.34K

Name           PID  CPU%  PgSp Owner
reaffin      524308  2.0  100K root            
java       48234514  1.3 1.12G sas             
topas_nm    8192046  0.5 7.44M root            
java       27918426  0.5  212M sas             
nfssync_    2293778  0.3  108K root            
java       14876674  0.3  209M sas             
topas      19529824  0.2 5.27M root            
dtterm     26148896  0.2 1.58M root            

You can see the line for the waitstates; at the moment I took this snapshot, the server was quite idle.

[Astounding]

There are also the typical issues to consider (not part of your original question).

Is one program tying up a data set that the other program requires?  (Sorting a data set usually ties it up.)

Does an output of one program become an input to another program?

[Kurt_Bremser]

---

@texasmfp wrote:

In particular, a 16-core CPU that is only using 4% of its capacity on the first program. 

---

Which means you already have a problem. 4% is about half a core, very little in light of many SAS procedures being multi-threaded. This points to a lot of I/O waits or delays caused by network transfers (are you reading from a remote DBMS?).

You need to inspect your jobs in this regard.