Hi:
This is not a simple question. You need to know how your data will be accessed for INPUT, what kind of processing will be performed, what kind of transformations (if any) have to take place and what kinds of OUTPUT (either output data or output report) need to be produced.
For example, is your data in "flat files" or ASCII text files? Is your data in Oracle or DB2 tables? Is your data in SQL server files? Does the data live on the same platform/operating system as SAS? Are you using a DATA step? Are you using PROC SQL? Is SAS/Access to Oracle or SAS/Access to DB2 involved? Is your data in Teradata or SAP or SPD Server? What is "huge"? A million rows? Or is it only 200,000 rows, but very wide?? Or is the data both very wide (lots of variables) and very long (lots of rows)?
There's no simple answer to this question, because the answer is VERY specific to your configuration, where your INPUT data lives, the limitations of the operating system, the limitations of the network, the kind of processing you need to do, the kind of output you need to produce, etc, etc. You will always have to live within constraints
-- operating system constraints: like CPU cycles and I/O operations;
-- space constraints: physical storage constraints and work area constraints;
-- time constraints: time to load, time to process, time to transmit across a network; and
-- manpower constraints: time to maintain and or recode programs.
If you optimize for one constraint, you could end up having a negative impact on another constraint.
If you go to Google and enter:
SAS accessing large data
or
SAS efficient access data
in the search box, you will find some hits for SUGI papers that talk about efficiencies for accessing large data files. This would be a good place to start your investigations.
cynthia
