topic Techniques that you apply to huge data (approx 1-2 TB) for data modelling. in SAS Programming

Techniques that you apply to huge data (approx 1-2 TB) for data modelling.

TSarthaka2018 — Fri, 10 Jan 2025 06:01:51 GMT

Hi, I have about 2 TB of data that requires data prep and other operation and eventually leading to Model development using Linear Regression.

Please help me with all the procedures that you use to process such huge data while saving time.

Re: Techniques that you apply to huge data (approx 1-2 TB) for data modelling.

Ksharp — Fri, 10 Jan 2025 09:17:45 GMT

You could check any PROC start with HP .
Like:
PROC HPGENSELECT
PROC HPLOGISTIC

Re: Techniques that you apply to huge data (approx 1-2 TB) for data modelling.

LinusH — Fri, 10 Jan 2025 07:14:23 GMT

If this is one off, I don't think it doesn't matter so much.
If this is a recurring work that should be executed regularly - try to do data prep "and other operations" on only new data.

Re: Techniques that you apply to huge data (approx 1-2 TB) for data modelling.

SASKiwi — Fri, 10 Jan 2025 22:31:03 GMT

Take a random sample of your data and do all of your data preparation and modelling development on that. Once you have got it working OK, then try it on the full data.