@RW9 wrote: Why are the data stored like that? From a storage point of view having multiple tables with similar data is inefficient - you have the extra overhead not only of the header block, but a read/write on each. From a programming point of view it is inefficient - as you are finding now, trying to code with such a setup is a nightmare, creating masses of messy macro code. Imagine how much simpler life would be if you stored all that data in one big dataset, with a column (you know those parts of the datasets used to capture data) for date, then your code would be as simple as: data want;
set have;
where date between today() and intnx('month',today(),-1);
run; Its your data modelling causing your issues - and most likely issues across the board. Sadly space and processing power is a problem. We are talking about 100's of millions of records. So having it all in one big table makes it near impossible to statistical analysis on the data on a monthly basis let alone even more detailed disaggregated analysis on region, product, or day level So its broken up into smaller pieces (i.e monthly data). But im busy developing an editing system to flag outlying data so need a proper time series (based on user inputs) and this lead me to this.
... View more