topic Re: One big data set or multiple data sets in SAS Programming

One big data set or multiple data sets

Ronein — Mon, 06 Aug 2018 10:05:17 GMT

Hello

I want to ask a question about efficiency.

In my work every month a data set (raw data) of Loans balances is created.

What is more efficiency way:

option1- Every month create another raw data data set

option2- Every month add new rows to the previous month data set.

What do you think is a better way to do it?

Why?

You need to know that lots of queries will be written based on these raw data tables.

Thanks

Joe

Re: One big data set or multiple data sets

Kurt_Bremser — Mon, 06 Aug 2018 10:11:31 GMT

If your queries/reports are mainly run for single months, it makes sense (performancewise) to keep the datasets separate.

Re: One big data set or multiple data sets

RW9 — Mon, 06 Aug 2018 10:13:36 GMT

Splitting same data is rarely a good idea. Think of it this way, if you have multiple datasets split by date you:

1) have lots more files with their own header block - takes more space

2) have to write messy code to process lots of different dataset names - takes more time an resources to do so and is less stable

3) have code which is opening, closing, writing etc. data lots of times, inefficient for processing power

Hence the conclusion I would come to is no, don't split the data out into sections, have one single table with a fixed simple structure which is very easy to program against, query etc. Just look at how the big databases do it, Oracle has a few tables with simple structures to hold lots of data.

As a bit of a disclaimer, this is general, if your looking at something more specific like big data, or disparate data then there are other models for such a thing. But in general, for most tasks, keep it simple, keep it generic.