Hi,
I have written a code which is processing 60 million records and taking around ~ 1.5 hrs to run.
There is a push for further reducing it to 15 mins.
The data set has only text columns and there are 15 columns in the data set.
I need some expert advice whether SAS can process 60 M records in 15 mins or not.
Regards,
Anurag Rai
It can. Show us your code if you want advise.
hi,
Thanks for the prompt reply!!!
There are about 30 input file which are acting as a data source to the code.
There are about 250 Business Rules which are kind of if else condition created as different data sets than finally getting appended.
Regards,
Anurag Rai
Is the dataset stored with compress=yes? This is often a very big factor when dealing with character variables.
And show us the code in question.
To answer your question, the answer is "yes, surely". It is a matter of making your data structures and code efficient, and sizing the infrastructure (CPU, RAM, storage) correctly.
hi,
Thanks for the prompt reply!!!
No it is not stored by using compressed option.
Regards,
Anurag Rai
@anuragraishines wrote:
hi,
Thanks for the prompt reply!!!
No it is not stored by using compressed option.
Regards,
Anurag Rai
Then you must test it with the compress option. This may reduce physical dataset file size significantly and therefore your I/O load.
Also run your steps with options fullstimer to get more in-depth diagnostic information. A large difference between CPU time and real time always points to the I/O subsystem.
You may also want to include an environment description. If the data manipulation involves network storage/transmission then you may have network issues to consider. If the data is read/written to another database system you connection may have an impact.
hi,
Thanks for the prompt reply!!!
There are about 30 input file which are acting as a data source to the code.
There are about 250 Business Rules which are kind of if else condition created as different data sets than finally getting appended.
And it is not Viya platform.
Regards,
Anurag Rai
Join us for SAS Innovate 2025, our biggest and most exciting global event of the year, in Orlando, FL, from May 6-9. Sign up by March 14 for just $795.
SAS' Charu Shankar shares her PROC SQL expertise by showing you how to master the WHERE clause using real winter weather data.
Find more tutorials on the SAS Users YouTube channel.