BookmarkSubscribeRSS Feed
Ronein
Meteorite | Level 14

Hello

I want to ask a question about efficiency.

In my work every month a data set (raw data) of Loans balances is created.

What  is more efficiency way:

option1- Every month create another raw data data set 

option2- Every month  add new rows  to the previous month data set.

What do you think is a better way to do it?

Why?

You need to know that lots of queries will be written based on these raw data tables.

 

 

Thanks

Joe

2 REPLIES 2
RW9
Diamond | Level 26 RW9
Diamond | Level 26

Splitting same data is rarely a good idea.  Think of it this way, if you have multiple datasets split by date you:

1) have lots more files with their own header block - takes more space

2) have to write messy code to process lots of different dataset names - takes more time an resources to do so and is less stable

3) have code which is opening, closing, writing etc. data lots of times, inefficient for processing power

 

Hence the conclusion I would come to is no, don't split the data out into sections, have one single table with a fixed simple structure which is very easy to program against, query etc.  Just look at how the big databases do it, Oracle has a few tables with simple structures to hold lots of data.

 

As a bit of a disclaimer, this is general, if your looking at something more specific like big data, or disparate data then there are other models for such a thing.  But in general, for most tasks, keep it simple, keep it generic.

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 2 replies
  • 569 views
  • 1 like
  • 3 in conversation