Desktop productivity for business analysts and programmers

Importing a very large .txt file

Accepted Solution Solved
Reply
Occasional Contributor
Posts: 8
Accepted Solution

Importing a very large .txt file

Hello,

 

I am a new user working with SAS Enterprise Guide and I have been trying to import a very large .txt file, with over 10 million rows and around 18 columns.

I have tried the wizard, which does not bring all the records, so I tried the proc import:

 

PROC IMPORT
    DATAFILE='\\10.42.16.29\gc$\Full_0151108.txt'
    OUT = gc.20151108TESTE;
    DBMS = DLM replace;
    DELIMITER = '|';
    getnames=yes;
run;

 

The code imports fewer lines than the wizard. I can't seem to find a way to import all of the rows! Any help appreciated.

 

Thanks.Smiley Frustrated

 

 


Accepted Solutions
Solution
‎12-30-2015 04:47 AM
Respected Advisor
Posts: 4,812

Re: Importing a very large .txt file

Try adding option IGNOREDOSEOF to your infile statement.

PG

View solution in original post


All Replies
Occasional Contributor
Posts: 18

Re: Importing a very large .txt file

Hi,

 

Could you try following code:-

 

data work.temp ;
infile '<file_path>\abc.txt'
lrecl = 32767 <or whatever is your file length>
delimiter = '|'
dsd
missover
firstobs = 1 /*Give this value as 1 if your text file does not contain column name. If your file contains column name this value will be 2*/;
;
attrib <col1> length = $32;
attrib <col2> length = $8;
attrib <col3> length = $26;

 

/*This will continue till col 16*/

input <col1> <col2> <col3>......<col16>;

run;

Occasional Contributor
Posts: 8

Re: Importing a very large .txt file

Thanks, I have tried, but only imports the same as my code, 26357 records...
Solution
‎12-30-2015 04:47 AM
Respected Advisor
Posts: 4,812

Re: Importing a very large .txt file

Try adding option IGNOREDOSEOF to your infile statement.

PG
Occasional Contributor
Posts: 8

Re: Importing a very large .txt file


PGStats wrote:

Try adding option IGNOREDOSEOF to your infile statement.


Thansk PG, I did just that yesterday after I learned there were special characters on the txt file. It worked like a charm.

I had to get an IT guy to open the file for me because I dont have any tools to open such a big file, but when i tried to load another file and analysed the last line it imported and finding a very similar error, it was just a matter of finding what  was wrong with the file.

 

Smiley Happy

Super User
Posts: 19,063

Re: Importing a very large .txt file

When you use proc import you'll get some code in the log. Take that code and modify it to read your full file. Most likely you'll need to change the LRECL option because your lines are longer than SAS expects.
Super User
Posts: 9,865

Re: Importing a very large .txt file

What did your SAS LOG display ? Is there any WARNING or ERROR message? I also notice OUT = gc.20151108TESTE; in your code. It is not right. SAS table name can't be start with a digit .
Super User
Posts: 19,063

Re: Importing a very large .txt file


Ksharp wrote:
What did your SAS LOG display ? Is there any WARNING or ERROR message? I also notice OUT = gc.20151108TESTE; in your code. It is not right. SAS table name can't be start with a digit .

 

This was posted in SAS EG forum, and unfortunately SAS EG sets the option validvarname=any by default.

Super User
Posts: 9,865

Re: Importing a very large .txt file

I see. But there are must have something like NOTE: WARNING: in LOG. That could help us to know where is the problem, if OP think importing result is not right.
Occasional Contributor
Posts: 18

Re: Importing a very large .txt file

Hi,

 

Can you please paste few rows after 26,357? Also, while running the code did you find any erroneous statement in log?

 

 

🔒 This topic is solved and locked.

Need further help from the community? Please ask a new question.

Discussion stats
  • 9 replies
  • 649 views
  • 0 likes
  • 5 in conversation