Desktop productivity for business analysts and programmers

Importing a very large .txt file

Accepted Solution Solved
Reply
Contributor
Posts: 64
Accepted Solution

Importing a very large .txt file

I have a large, multi-GB .txt file to import into SAS. (my SAS is running on a remote server)

 

First, I FTP transferred my .txt file to the server itself.

 

Next, I launched the Import Wizard from my local E.G. client, but whats happening is   "Retrieving Source File" - which is silly because the data is already there. 

 

How can I import the .txt file most efficiently? I am aware of proc IMPORT but then I lose the ability to review the datatypes during the EG wizard. 


Accepted Solutions
Solution
‎01-22-2016 11:31 PM
Grand Advisor
Posts: 17,288

Re: Importing a very large .txt file

Is this a one time process?

 

In general, it's best to use a data step to read the file, but that means explicitly listing fields/types.

The 'lazy trick' is to use proc import, most likely with option obs= set to generate the code.

Take the code from the log, review and customize the code as desired and run from there.

 

option obs=100000;

proc import ...

run;

 

Grab code from log and you're good to go. You can use ALT+Click to select code without log numbers.

Press the ALT key and at the same time select the top left-hand corner of your log in the column just to the right of your line numbers by clicking with your mouse button. Now drag the mouse down to the bottom right-hand corner of your log. You should now have highlighted your SAS program without the line numbers.

Courtesy of @SASKiwi

https://communities.sas.com/t5/SAS-Procedures/How-to-remove-log-numbers/td-p/21489

 

View solution in original post


All Replies
Solution
‎01-22-2016 11:31 PM
Grand Advisor
Posts: 17,288

Re: Importing a very large .txt file

Is this a one time process?

 

In general, it's best to use a data step to read the file, but that means explicitly listing fields/types.

The 'lazy trick' is to use proc import, most likely with option obs= set to generate the code.

Take the code from the log, review and customize the code as desired and run from there.

 

option obs=100000;

proc import ...

run;

 

Grab code from log and you're good to go. You can use ALT+Click to select code without log numbers.

Press the ALT key and at the same time select the top left-hand corner of your log in the column just to the right of your line numbers by clicking with your mouse button. Now drag the mouse down to the bottom right-hand corner of your log. You should now have highlighted your SAS program without the line numbers.

Courtesy of @SASKiwi

https://communities.sas.com/t5/SAS-Procedures/How-to-remove-log-numbers/td-p/21489

 

Contributor
Posts: 64

Re: Importing a very large .txt file

Still a pain - that the client needs to bring back all the data like that -  but your "lazy" trick is incredible and I very much appreciate it! 

The added bonus is the Alt+Click trick, which I am sure will serve me well many times in my career.

 

Thanks!

Grand Advisor
Posts: 17,288

Re: Importing a very large .txt file

If your lucky Smiley Wink your client may have provided a data transfer agreement with the data specifications. If so, I usually find a way to create my informat/input statements from that. There's a variety of ways - usually I use Excel to build it since I can customize and make it semi-automated manner. 

Contributor
Posts: 64

Re: Importing a very large .txt file

No such luck. : ( What started off as a favor has turned into a full blown project. Thanks for the pro tips!


☑ This topic is SOLVED.

Need further help from the community? Please ask a new question.

Discussion stats
  • 4 replies
  • 361 views
  • 2 likes
  • 2 in conversation