How to deal with large dataset? Asking for tips

[Phil_NZ]

Hi all SAS experts,

After a while, I wrap up my code. The problem now I can see is how to deal with 1,2TB data splitted to 8 datasets.

I try to find some ways to deal with a large datasets, the one very impressive to me is to use view (Data step view). Can I ask from your experience is there anything else I should take a look at when dealing with a large dataset to save storage. I know there should be specific ways with specific codes, but there should be something, in general, I want to learn first.

Warmest regards.

[andreas_lds]

Maybe start by reading "High-Performance SAS Coding" by @ChrisNZ 

[ChrisNZ]

Indeed, @andreas_lds, indeed 🙂

[Ksharp]

The fastest way is :

data F M;
set sashelp.class;
select(sex);
when('F') output F;
when('M') output M;
otherwise;
end;
run;

[ChrisNZ]

>  is there anything else I should take a look at when dealing with a large dataset to save storage

That's a very vague question.

> there should be something, in general, I want to learn first.

You can look at options BUFNO, BUFSIZE, and SGIO (if you are using Windows) to speed up data transfer.

>  dealing with a large dataset to save storage

the SPDE engine with binary compression is very good at reducing file size, and therefore disk access. It brings other goodies too, like on-the-fly-sorting, or faster indexes.

[Kurt_Bremser]

First, we need to know what you want to do with that large dataset (which information you want to gain), so we can determine how to do it in a most efficient way.