Understanding the basics of how to determine what hardware component is bottlenecking your system

[jcsimmo]

Just some background I am a physician with a comparatively strong understanding of computers (enough to build my own rig: i7 8700K, 64 GB of ram, m.2 samsung SSD, nvidia gtx 1070) who is self-taught SAS. I run queries on a dataset that is 30GB located on my SSD and I guess would love someone to help me understand how I can determine if my hard drive is bottlenecking my system.

Just to be clear, I have updated my SAS config file to the following

-cpucount 6

-memsize 60G

-threads

-realmemsize 56G

-sumsize 60G

-sortsize 60G

When I look at the resource monitor, my CPU hovers around 11%, the RAM utilization is 10%, and the I/O speed on the disk is 30mb/sec. I looks like neither the CPU or RAM is limiting me but its hard to believe that an m2 hard drive is limiting me as well. I have two questions

• How can I determine what’s bottlenecking my system
• What type of hard drive benchmark (read, write, seq-read/write, 4k read, 4k write, 4k mixed) is most applicable to the performance of SAS.

[VDD]

I have bottleneck issue with SAS, but I have found if I run the process's on the SAS server the network bottleneck issue disappears.

This doesn't sound like it is your issue, but though I would mention it.

[SASKiwi]

Your SAS log is the first place to start if you think you have performance issues. Run a SAS program with the FULLSTIMER SAS option.

Compare the elapsed time of each of the steps with your CPU time. If your elapsed time is similar to your CPU time then that step doesn't have a bottleneck. If your elapsed time is significantly slower than your CPU time, then you may have a bottleneck. Check the memory usage of the step - is it anywhere near the physical limit? If not your most likely bottleneck is IO. You can prove this by say halving the number of rows in the dataset you are processing. Does it reduce the processing time by a similar amount?

Hopefully this will give you enough ideas to experiment with.

[Kurt_Bremser]

For today, 30 mb/s is very slow. The LVD SCSI disks on my 10-year old pSeries I decommissioned this spring would give me a minimum of 100 mb/s, even with multiuser/multiprocess load. There must be some configuration problem.

[jcsimmo]

Good to know. Certain SSD are optomized for certain tasks eg 4k write/right, seq read etc. Which task is most closely associated with use in SAS? As in, a hard drive that has the fastest ______ (eg. 4k write speed) will translate into the fastest processing of SAS data

[Kurt_Bremser · Accepted solution]

Out of experience, I'd go for the disks that give you the highest continuous write throughput. Most of the time, SAS does sequential reads and writes on datasets; random access happens mostly in utility files created when you do complicated SQL joins/operations.

SSDs have the big advantage that they don't create latencies when concurrent use (or SQL, as noted above) forces random accesses; if you can't satisfy your I/O needs with one drive, get several and create a striped volume over them. Also keep WORK/UTILLOC volumes physically separate from the rest of your storage.