topic Re: Very large data files in SAS Programming

Very large data files

makset — Thu, 13 Aug 2020 16:46:38 GMT

I have several data files about 1 TB in total (16 files).
I want to have access to the data contained therein as soon as possible. I am asking for some advice and thoughts.
my workstation:

W8.1Pro

sas 9.2

2x xeon e5-2630 v3

RAM 64GB

nvme samsung mzvpv256

nvme wds100t3xoc-00sj 1TB

Best regards and thank you in advance for your help

Re: Very large data files

Reeza — Thu, 13 Aug 2020 17:22:31 GMT

SAS processes data row by row so some things work quite well on a desktop regardless of size, but take time.

Are these SAS files or text files? And most importantly - what's your question? What do you need help with?

Re: Very large data files

makset — Thu, 13 Aug 2020 17:45:59 GMT

@Reeza wrote:
Are these SAS files or text files?

sas file

@Reeza wrote:
what's your question? What do you need help with?

The question is not about a specific problem.
I have, for example, 16 cores and a fairly fast nvme, so maybe I will divide each file into 16 equal parts and process them in parallel?
maybe there is another way?
maybe a hasch table? I'm learning now

best regards

Re: Very large data files

Tom — Thu, 13 Aug 2020 18:41:29 GMT

It all depends on what you are trying to do. In general it might be best to summarize the data into much smaller size and then do your analysis from the summary. But whether that is possible or how to do it depends on what analysis you are doing.

Re: Very large data files

Reeza — Thu, 13 Aug 2020 19:14:31 GMT

A vague, general question gets a vague general answer.

Re: Very large data files

SASKiwi — Thu, 13 Aug 2020 22:34:48 GMT

I doubt there is hash functionality in SAS 9.2.

I recommend you get your SAS software up-to-date. I would expect SAS 9.4 to be more efficient and offer more capabilities than 9.2.

Re: Very large data files

andreas_lds — Fri, 14 Aug 2020 05:13:33 GMT

I would start by upgrading to the latest sas version. Both nvme you mention are way to small to hold all dataset you want to process, so where is the data stored?

Re: Very large data files

makset — Fri, 14 Aug 2020 09:11:24 GMT

Migration to sas 9.4 (TS1M6) is not possible yet, I don't have time, maybe in October. Anyway, I was thinking to rewrite everything to c ++, but so far I don't have much experience with c ++.
system is on nvme samsung mzvpv256
All the data is stored here nvme wds100t3xoc-00sj 1TB, and I have one more

Re: Very large data files

makset — Fri, 14 Aug 2020 09:13:02 GMT

What in sas 9.4 (TS1M6) is so good (better than 9.2)

Re: Very large data files

makset — Fri, 14 Aug 2020 09:17:10 GMT

Write what comes to your mind in this specific topic or indicate places where I can find something valuable in your opinion and what I could have missed
-I used the hash table
-I got interested in sasphile but it has big limitations in my opinion

Re: Very large data files

SASKiwi — Sat, 15 Aug 2020 00:37:31 GMT

I suggest you refer to the SAS documentation for a complete list of improvements (documentation.sas.com). There are hundreds if not thousands of them.

I'd expect SAS 9.4 to be significantly faster than 9.2 as well, although that may depend on the type of processing you are doing.

BTW I just checked and you are in luck. DATA Step HASH was implemented in SAS 9.1 so you will have it in 9.2: https://support.sas.com/kb/11/391.html

Re: Very large data files

SASKiwi — Sat, 15 Aug 2020 00:28:29 GMT

IMO,it sounds like you don't have enough storage regardless of what language you use to process your data files. If your 16 files completely fill your disk drive you have no room to do anything else.

Re: Very large data files

mkeintz — Sat, 15 Aug 2020 05:27:33 GMT

If I understand your spec's correctly you have 1.25TB of available storage in which you want to store and use some number of SAS datasets the you say adds up to 1TB.

Depending on what you want to do with the datasets, you might run out of disk space. Say one of the datasets is .2TB, and you want to sort it. That process, at some point, would need about .2TB for intermediate sort utility files, and .2TB for the new sorted dataset prior to deleting the intermediates. I.e. you could easily have a point at which you require 1.4TB.

Aside from sorting issues, in the case of many SAS programs, users often generate lots of temporary sas datasets (in the work libname, if nowhere else), over the course of a multi-step program. I can easily see the possibility of running into disk space limitations without conducting more than casual housekeeping.

Re: Very large data files

mkeintz — Sat, 15 Aug 2020 05:32:01 GMT

@SASKiwi

hash functionality has been there longer than you might think.

According to The SAS Hash Object in Action:

In SAS® Version 9.1, the hash table - the very first object introduced via the DATA Step Component Interface in Version 9.0 - has finally become robust and syntactically stable

Re: Very large data files

SASKiwi — Sat, 15 Aug 2020 08:08:29 GMT

@mkeintz - Yes, see my follow up post.

Re: Very large data files

makset — Sat, 15 Aug 2020 11:42:29 GMT

I'm a bit sorry because you're focusing on what's the least important (but I understand it's important). I still don't have 1TB of data, but soon. I will just buy the next nvme.

Re: Very large data files

SASKiwi — Sun, 16 Aug 2020 00:10:07 GMT

@makset - If we are giving you the wrong answers it is because you haven't fully explained what your problem is. You've listed your PC hardware and explained the size of data you have but you haven't said anything about what you want to do with it. Until you explain what data processing you want to do then we can't provide better advice. BTW hashing isn't a magic wand to improve all types of processing performance.

Processing 1TB of data on a PC with SAS is entirely possible and many PC SAS users have bigger data volumes. As already explained you will probably need at least 2 or 3TB of free space for further processing as well as 1TB just for your source files. How much extra depends on what type of processing you want to do.

Re: Very large data files

Patrick — Thu, 20 Aug 2020 21:16:03 GMT

@makset

Your question is way to vague and generic to propose anything else than reading/books. So if it's about dealing with large data sets then you might want to Google for SAS topics dealing with performance - and there's lot about this out there.

Also spending time reading SAS® 9.4 Language Reference: Concepts, Sixth Edition is likely valuable for you as it explains how SAS actually works and how it processes data.

About SAS hashes: They get loaded into memory with fully expanded column length so likely not suitable for your huge tables unless you've got actually the required memory available.