keeping runing process when doing PCA analysis with large dataset of microbial ASV data

[Tom]

---

@kellychan84 wrote:

Hello Tom,

I don't know why it shows only 34 observations. After I add the "dsd truncover lrecl=1000000" coding, the column number finally increases from 5000 to exactly what I have (5359). Paige's proc pls procedure is giving me results finally!!

---

Not sure why the file should be truncated at 34 lines. Make sure your file isn't corrupted.  Do you have a backup?

You might try writing the data to a new file and compare the sizes.  If the operating system reports the new file is much smaller than the original then perhaps there are more lines in the original and something made SAS stop at 34 lines.

You could try adding the IGNOREDOSEOF option to the INFILE statement and see if there is a ctrl-Z in the file that is causing SAS to stop early.

[kellychan84]

Hello Tom, it seems it is not working, already run for 20 mins.

[sbxkoenk]

Hello,

I am sure SAS allows more than 1 million variables (columns).

There's no reason your dataset is limited to 5000 instead of 5359.

Maybe you need to specify :

LRECL=logical-record-length

on infile statement. 

LRECL= specifies the physical line length of the file. LINESIZE= tells the INPUT statement how much of the line to read.

Cheers,

Koen

[PaigeMiller]

---

@kellychan84 wrote:
I only run the first input step. it shows 5000 column, but may data set has 5359 column. That means something wrong here and can not proceed?

---

That does not mean anything is wrong. 

[ballardw]

---

@kellychan84 wrote:
I only run the first input step. it shows 5000 column, but may data set has 5359 column. That means something wrong here and can not proceed?

---

Show us the LOG from running that data step. Copy the text from the LOG with all the code, notes, and messages including how many variables and observations. On the forum open a text box using the </> icon and paste the text.

The text box is to preserve formatting of the log text.

You may need to increase the LRECL setting on the INFILE statement. The default of 32767 may not be long enough to read all the characters plus delimiters on a data set with 5360 variables (you didn't count TREATMENT). I would try adding LRECL=50000 to the Infile and see if you get more variables.

[Tom]

---

@kellychan84 wrote:
Hello Paige, it is 182,000 observations with 5359 variables. Even the proc print step, it just keep running without stop.

---

Do NOT run PROC PRINT on such a large dataset. Are you actually going to read all 57 gazillion pages that would produce?

[kellychan84]

At first, I did not run print, I add it later then to check are there something at the first input step. now I remove it. Thank you!