Some physical questions:
1/ You are saying these are SAS datasets not externally stored ones in a RDBMS?
1,5Tb is big not that big try to gather information on the physical constraints on the IO-system.
Spreading IO processing on several physical will gain in II time. Are there SSD's involved?
2/ You system will have some CPU's and memory. You will need to know what you are having and how it can be used.
Spreading load over cpu's will decrease total processing time but possible overload the system
Using the memory as much as possible instead of IO wel decrease processing time. Overloading memory can cause trashing by wich you loose any progress.
3/ SAS system options.
These can improve processing by optimizing blocksizes (not too small) set those according OS system setting (aligniofiles) minimizing conversion overheads.
Compressing (compress= ) can help at the cost of CPU minimizing IO. This is dependent on the datastructures and contents.
Within this area it is the goal to optimize utilization.
Some logical analytics approaches:
1/ get rid of necessary data as quick as possible and proceed with as small as possible
2/ minimize necessary steps in your processing. Better run at once on data than accessing that over and over again.
For example A SAS dataset can split I multiple in one run using a sas datastep. Using Sql that needs several passes.
3/ The effects of physical sequential IO or random access IO. The first one will run much faster (head movements harddisk).
Some rule of thumb: When you are accessing more as 20% of the data you can process it sequentially. When less random access will be faster.
When some datasets are accessed random more often define indexes on them (or direct pointers)
Your ideas:
Sorting... Defining indexes could eliminate that. (think on 20% thumb rule)
proc means ... Can run sequential all analyses done as class statements. Instead of running multiple times one pass.
The disadvantage will be thinking on memory usage memsize setting etc.
merge That is often a standard approach for table-lookups. for table-lookups think on the hash object and using formats.
Those are cpu/memory oriented no IO constraints. Doing the job better than the SQL only thinking
(much more)
coding designed for big data processing will also work on small data. Test and validate your code with small sizes (sample or set obs=)
coding just for small data does not need to able run on big data. (constraints limitations). Designing your process and rethinking the performance aspects is an approach to avoid pitfalls. How often do we hear: it run on my sample but not on the real data.
---->-- ja karman --<-----
