topic Re: How to deal with large dataset? Asking for tips in SAS Programming

How to deal with large dataset? Asking for tips

Phil_NZ — Thu, 11 Mar 2021 09:50:17 GMT

Hi all SAS experts,

After a while, I wrap up my code. The problem now I can see is how to deal with 1,2TB data splitted to 8 datasets.

I try to find some ways to deal with a large datasets, the one very impressive to me is to use view (Data step view). Can I ask from your experience is there anything else I should take a look at when dealing with a large dataset to save storage. I know there should be specific ways with specific codes, but there should be something, in general, I want to learn first.

Warmest regards.

Re: How to deal with large dataset? Asking for tips

andreas_lds — Thu, 11 Mar 2021 10:59:54 GMT

Maybe start by reading "High-Performance SAS Coding" by @ChrisNZ

Re: How to deal with large dataset? Asking for tips

Ksharp — Thu, 11 Mar 2021 11:02:11 GMT

The fastest way is :

data F M;
set sashelp.class;
select(sex);
when('F') output F;
when('M') output M;
otherwise;
end;
run;

Re: How to deal with large dataset? Asking for tips

ChrisNZ — Thu, 11 Mar 2021 22:08:29 GMT

> is there anything else I should take a look at when dealing with a large dataset to save storage

That's a very vague question.

> there should be something, in general, I want to learn first.

You can look at options BUFNO, BUFSIZE, and SGIO (if you are using Windows) to speed up data transfer.

> dealing with a large dataset to save storage

the SPDE engine with binary compression is very good at reducing file size, and therefore disk access. It brings other goodies too, like on-the-fly-sorting, or faster indexes.

Re: How to deal with large dataset? Asking for tips

ChrisNZ — Thu, 11 Mar 2021 22:09:15 GMT

Indeed, @andreas_lds, indeed 🙂

Re: How to deal with large dataset? Asking for tips

Kurt_Bremser — Thu, 11 Mar 2021 22:42:55 GMT

First, we need to know what you want to do with that large dataset (which information you want to gain), so we can determine how to do it in a most efficient way.