topic Re: how to pull huge data using sas programming in SAS Health and Life Sciences

how to pull huge data using sas programming

deleted_user — Wed, 29 Aug 2007 17:37:29 GMT

hi
my question is ,now to pull huge data using sas programming.

Re: how to pull huge data using sas programming

Cynthia_sas — Thu, 30 Aug 2007 02:51:03 GMT

Hi:
This is not a simple question. You need to know how your data will be accessed for INPUT, what kind of processing will be performed, what kind of transformations (if any) have to take place and what kinds of OUTPUT (either output data or output report) need to be produced.

For example, is your data in "flat files" or ASCII text files? Is your data in Oracle or DB2 tables? Is your data in SQL server files? Does the data live on the same platform/operating system as SAS? Are you using a DATA step? Are you using PROC SQL? Is SAS/Access to Oracle or SAS/Access to DB2 involved? Is your data in Teradata or SAP or SPD Server? What is "huge"? A million rows? Or is it only 200,000 rows, but very wide?? Or is the data both very wide (lots of variables) and very long (lots of rows)?

There's no simple answer to this question, because the answer is VERY specific to your configuration, where your INPUT data lives, the limitations of the operating system, the limitations of the network, the kind of processing you need to do, the kind of output you need to produce, etc, etc. You will always have to live within constraints
-- operating system constraints: like CPU cycles and I/O operations;
-- space constraints: physical storage constraints and work area constraints;
-- time constraints: time to load, time to process, time to transmit across a network; and
-- manpower constraints: time to maintain and or recode programs.

If you optimize for one constraint, you could end up having a negative impact on another constraint.

If you go to Google and enter:
SAS accessing large data
or
SAS efficient access data
in the search box, you will find some hits for SUGI papers that talk about efficiencies for accessing large data files. This would be a good place to start your investigations.

cynthia

Re: how to pull huge data using sas programming

deleted_user — Thu, 13 Sep 2007 03:00:35 GMT

How huge is your SAS datasets in clinical trials?

Generally there's no huge data in clinical trials! The data structure is complex or messy, but definitely not huge...

A standard Phase III study, 100 patient, 300 pages each patient, 20 data points on every page...

Re: how to pull huge data using sas programming

deleted_user — Thu, 04 Oct 2007 16:08:58 GMT

I have a similar question. I am supposed to extract data from eDataFax using SAS. Any suggestions? I never use eDataFax before. Thanks a bunch!

Re: how to pull huge data using sas programming

Cynthia_sas — Thu, 04 Oct 2007 19:01:05 GMT

Hi,
Apparently, this is a topic that DataFax discusses in their documentation and other users appear to have written papers about -- a quick Google search revealed these links:
SAS-L Posting: http://listserv.uga.edu/cgi-bin/wa?A2=ind0403a&L=sas-l&F=&S=&P=8582 message recommends reading "DataFax Programmer Guide, Chapter 10 DFsas: DataFax to SAS. The chapter is about 30 pages long".

http://www.datafax.com/UserGroup/2002/presentations/Harris.pdf
http://www.datafax.com/UserGroup/2006/presentations/richardson.pdf
http://www.datafax.com/UserGroup/2007/presentations/ZO.pdf

http://www.basas.com/pages/baysas.htm presentation "Single Step Processing, DataFax to SAS Datasets in a Windows Environment (Simplifying Your End User Interface)", by Brad Harris

cynthia