Hello,
i'm currently in a migration moving from AIX to UNIX (RHEL) and we need to migrate our sas dataset files from source to new server, since there is an incopatibility between filesystems, and we're using PROC MIGRATE mapping source libraries on new server RHEL, PROC MIGRATE runs about an average of 150GB per PROC running ( i've seen i can run more than 5 PROCs at the same time without problem) and this tooks arround 10-20% of the cpu consumption, my question, is there any posibility to increase the ratio or the resurces the PROC takes to increase the performance and get better migration rates? my goal would be 300GB/h in order to have all the data migrated in 1 day.
If any one has encounter this situation during a migration may can help!
Thanks
I take it that you have to move your data over the network. 300 GB/h translates roughly to 85 MB/s, which means you need a 1 Gbit network bandwidth throughout the whole distance, including all switches/firewalls.
I suspect that the MIGRATE procedure is single-threaded, which would explain the limitation.
Have you tried this:
SAS/CONNECT and PROC UPLOAD is another way of migrating between SAS installations. You do need the product installed and licensed on both SAS installations though. Also there are the Base SAS CPORT / CIMPORT procedures which can do whole SAS libraries in one process.
I would set up a method to copy (sftp) several files in parallel, and then (also parallel) run data steps to do the conversion.
Before that, run tests to see if CEDA (using the files as is) is feasible in production use.
Definitely worth exploring parallel processing in that case.
@W1ndwaker wrote:
I supose bandwith is not a problem, since i got 6 simultaneous proces running giving a maximum 720GB/h at peak performance, the problem comes when a single directory (or library as you wanna call it) has a big ammount of data like 2TB of data, then I can only run 1 proc migrate on that folder and this is where the bottleneck is for me, if i can get a single proc migrate to a 300GB would be great for folders about 2.5TB
Thanks for the reply btw!!
Sound like the bottle neck is the speed of reading from that location. Or perhaps more likely the problem is you are writing them all to same target location, as writing takes more time that reading since you get no boost from cacheing. Can you split up the files into batches and write to different physical output disks?
The SAS Users Group for Administrators (SUGA) is open to all SAS administrators and architects who install, update, manage or maintain a SAS deployment.
SAS technical trainer Erin Winters shows you how to explore assets, create new data discovery agents, schedule data discovery agents, and much more.
Find more tutorials on the SAS Users YouTube channel.