topic Re: keeping runing process when doing PCA analysis with large dataset of microbial ASV data in Statistical Procedures

keeping runing process when doing PCA analysis with large dataset of microbial ASV data

kellychan84 — Fri, 11 Feb 2022 16:49:56 GMT

Hello,

I am using the following codes to do PCA analysis with large dataset.

data beta_diversity;
  length treatment $20;
  Infile "/home/u39233094/sasuser.v94/Thesis/CSV file/Cecal beta-diversity for SAS PCA delete.csv" dlm="," firstobs=2;
  input treatment$ ASV1-ASV5359;
run;
Proc print data=beta_diversity;
run;
ods graphics on;
proc princomp data=beta_diversity         /* use N= option to specify number of PCs */
              STD               /* optional: stdize PC scores to unit variance */
              out=PCAout         /* only needed to demonstate corr(PC, orig vars) */
              plots=(scree profile pattern score);
var _numeric_;  /* or use _NUMERIC_ */
ods output Eigenvectors=EV;  /* to create loadings plot, output this table */
run;
proc sgplot data=PCAout aspect=1;
   scatter x=prin1 y=prin2 / group=treatment;
   xaxis grid label="PC1 (%)";
   yaxis grid label="PC2 (%)";
run;

My data matrix is like this:

The SAS studio kept running and did not show results. I don't know what is wrong with the procedures. Could anyone give me some hints on this. Thank you very much in advance!!

Re: keeping runing process when doing PCA analysis with large dataset of microbial ASV data

AMSAS — Fri, 11 Feb 2022 17:16:55 GMT

First identify which step is resulting in SAS Studio not responding
Which Data Step / PROC is running and not responding
Have you tried running it step by step (run the data step, then the PROC PRINT)

Re: keeping runing process when doing PCA analysis with large dataset of microbial ASV data

sbxkoenk — Fri, 11 Feb 2022 17:25:50 GMT

Hello,

I don't think a PROC PRINT is a good idea on a dataset with thousands of columns and possibly millions of rows.

Writing this output to the Results window in HTML format takes ages.

Browse your data in a data pane (from the library pane) right after the data-step.

Or use the obs= data set option in PROC PRINT.
Like :

Proc print data=beta_diversity(obs=10); run;

Cheers,

Koen

Re: keeping runing process when doing PCA analysis with large dataset of microbial ASV data

ballardw — Fri, 11 Feb 2022 17:28:48 GMT

How large is "large"? As in how many observations are in your data?

How long did you allow the program to run?

If your data is large you might well want to remove that Proc print. Just creating the output table for a large data set can take a lot of time and resources. Plus if it is that "large" what do you get out of looking at printed output?

Which piece "kept running"? Run one procedure or data step at a time.

You may need to go to this link https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.4/statug/statug_princomp_details05.htm and check on the computation resources.

The above will you show you how much memory may be needed. If the princomp procedure needs more memory than you have in your system then it may be spending a lot of time writing data used for computations to disk and then rereading it.

Also there is a formula showing the relationship of variables and records to time. Not that time goes up to calculate the correlation matrix, very roughly, as the square of the number of variables and for eigenvalues the cube of the number of variables. So with 359 variables I am not surprised that there might be some time to run.

Re: keeping runing process when doing PCA analysis with large dataset of microbial ASV data

PaigeMiller — Fri, 11 Feb 2022 17:46:56 GMT

How many observations? How many variables?

If you just want to plot PRIN1 and PRIN2, then you can run PROC PRINCOMP with the option N=2, which may take a lot less time depending on how the algorithm is coded in SAS.

Re: keeping runing process when doing PCA analysis with large dataset of microbial ASV data

kellychan84 — Fri, 11 Feb 2022 17:49:58 GMT

Thank you for your reply. I will check the computaion resources later. I calculate there are 182,000 observations of my data. Beause SAS studio is easy to quit and each time I let it run until the paltform quits. Even the print step takes like forever. Will it be better to use SAS PC version to run this kind of large data set?

Re: keeping runing process when doing PCA analysis with large dataset of microbial ASV data

kellychan84 — Fri, 11 Feb 2022 17:53:01 GMT

Hello Paige, it is 182,000 observations with 5359 variables. Even the proc print step, it just keep running without stop.

Re: keeping runing process when doing PCA analysis with large dataset of microbial ASV data

PaigeMiller — Fri, 11 Feb 2022 17:58:49 GMT

As others have said, it is pointless (and very time consuming) to do a PROC PRINT on this data. Just remove PROC PRINT from your code and run it again.

Re: keeping runing process when doing PCA analysis with large dataset of microbial ASV data

Tom — Fri, 11 Feb 2022 18:00:37 GMT

@kellychan84 wrote:
Hello Paige, it is 182,000 observations with 5359 variables. Even the proc print step, it just keep running without stop.

Do NOT run PROC PRINT on such a large dataset. Are you actually going to read all 57 gazillion pages that would produce?

Re: keeping runing process when doing PCA analysis with large dataset of microbial ASV data

kellychan84 — Fri, 11 Feb 2022 18:01:55 GMT

Hello Koen, Evan in 10 observations, it did not work. I try 10 mins. I don't know why? If I input only 10 variables of the data set, it comes out quickly.

Re: keeping runing process when doing PCA analysis with large dataset of microbial ASV data

kellychan84 — Fri, 11 Feb 2022 18:03:45 GMT

I only run the first input step. it shows 5000 column, but may data set has 5359 column. That means something wrong here and can not proceed?

Re: keeping runing process when doing PCA analysis with large dataset of microbial ASV data

kellychan84 — Fri, 11 Feb 2022 18:05:14 GMT

At first, I did not run print, I add it later then to check are there something at the first input step. now I remove it. Thank you!

Re: keeping runing process when doing PCA analysis with large dataset of microbial ASV data

Tom — Fri, 11 Feb 2022 18:16:03 GMT

Are you sure you told SAS to read the complete lines of data?

Check the notes in SAS log, it will show the minimum and maximum line lengths read from the file.

Note that the default line length INFILE will use is currently 32,767 bytes. If your lines are longer than that then you need to add the LRECL= option to your data step. There is not any real downsize to setting it longer (other than your data step might require a little more memory while it is running).

While you are at it add other common sense options like DSD (so that missing values and values with commas are treated properly) and TRUNCOVER (so INPUT does not move to a new line if there are too few values on the line.)

data beta_diversity;
  length treatment $20;
  infile "/home/u39233094/sasuser.v94/Thesis/CSV file/Cecal beta-diversity for SAS PCA delete.csv" 
    dlm="," dsd truncover lrel=1000000 firstobs=2
  ;
  input treatment ASV1-ASV5359;
run;

Re: keeping runing process when doing PCA analysis with large dataset of microbial ASV data

sbxkoenk — Fri, 11 Feb 2022 18:19:46 GMT

Hello,

I am sure SAS allows more than 1 million variables (columns).

There's no reason your dataset is limited to 5000 instead of 5359.

Maybe you need to specify :

LRECL=logical-record-length

on infile statement.

LRECL= specifies the physical line length of the file. LINESIZE= tells the INPUT statement how much of the line to read.

Cheers,

Koen

Re: keeping runing process when doing PCA analysis with large dataset of microbial ASV data

PaigeMiller — Fri, 11 Feb 2022 18:23:47 GMT

@kellychan84 wrote:
I only run the first input step. it shows 5000 column, but may data set has 5359 column. That means something wrong here and can not proceed?

That does not mean anything is wrong.

Re: keeping runing process when doing PCA analysis with large dataset of microbial ASV data

PaigeMiller — Fri, 11 Feb 2022 18:43:38 GMT

@kellychan84 wrote:
Hello Koen, Evan in 10 observations, it did not work. I try 10 mins. I don't know why? If I input only 10 variables of the data set, it comes out quickly.

As I said earlier, try running the PROC PRINCOMP with the option N=2 (or N=5 or whatever), you don't need all 5359 dimensions.

Re: keeping runing process when doing PCA analysis with large dataset of microbial ASV data

kellychan84 — Fri, 11 Feb 2022 19:02:35 GMT

I change the code according to your suggestions. it is still running.

Re: keeping runing process when doing PCA analysis with large dataset of microbial ASV data

PaigeMiller — Fri, 11 Feb 2022 19:06:24 GMT

That should be quicker, but I still don't know how long it will take to compute N=2 dimensions for 180,000 records and 5359 variables.

Re: keeping runing process when doing PCA analysis with large dataset of microbial ASV data

kellychan84 — Fri, 11 Feb 2022 19:21:18 GMT

Hello Tom, it seems it is not working, already run for 20 mins.

Re: keeping runing process when doing PCA analysis with large dataset of microbial ASV data

ballardw — Fri, 11 Feb 2022 19:22:47 GMT

@kellychan84 wrote:
. Even the print step takes like forever. Will it be better to use SAS PC version to run this kind of large data set?

Why are you bothering to print it at all? 180,000 lines with everything fitting on one line would be something like 2,250 pages. When add in 400 variables that means each line is likely only showing 12 or so values, so you with 5000+ (on rereading your code) variables each observation of the data set would take about 447 lines to display, times the 2250 single lines that would be on the order of a document with maybe a million or more pages. You do not read that sort of stuff. Get rid of the proc print. If the only purpose of the proc print was to verify that the data was read use the OBS= data set option to display maybe 10 observations

Proc print data=beta_diversity (obs=10) ;
run;

But I think the 5000 variables in Princomp still takes a lot of time.