topic Importing a very large .txt file in SAS Enterprise Guide

Importing a very large .txt file

JBerry — Sat, 23 Jan 2016 01:32:18 GMT

I have a large, multi-GB .txt file to import into SAS. (my SAS is running on a remote server)

First, I FTP transferred my .txt file to the server itself.

Next, I launched the Import Wizard from my local E.G. client, but whats happening is "Retrieving Source File" - which is silly because the data is already there.

How can I import the .txt file most efficiently? I am aware of proc IMPORT but then I lose the ability to review the datatypes during the EG wizard.

Re: Importing a very large .txt file

Reeza — Sat, 23 Jan 2016 04:22:34 GMT

Is this a one time process?

In general, it's best to use a data step to read the file, but that means explicitly listing fields/types.

The 'lazy trick' is to use proc import, most likely with option obs= set to generate the code.

Take the code from the log, review and customize the code as desired and run from there.

option obs=100000;

proc import ...

run;

Grab code from log and you're good to go. You can use ALT+Click to select code without log numbers.

Press the ALT key and at the same time select the top left-hand corner of your log in the column just to the right of your line numbers by clicking with your mouse button. Now drag the mouse down to the bottom right-hand corner of your log. You should now have highlighted your SAS program without the line numbers.

Courtesy of @SASKiwi

https://communities.sas.com/t5/SAS-Procedures/How-to-remove-log-numbers/td-p/21489

Re: Importing a very large .txt file

JBerry — Sat, 23 Jan 2016 04:30:58 GMT

Still a pain - that the client needs to bring back all the data like that - but your "lazy" trick is incredible and I very much appreciate it!

The added bonus is the Alt+Click trick, which I am sure will serve me well many times in my career.

Thanks!

Re: Importing a very large .txt file

Reeza — Sat, 23 Jan 2016 04:43:01 GMT

If your lucky 😉 your client may have provided a data transfer agreement with the data specifications. If so, I usually find a way to create my informat/input statements from that. There's a variety of ways - usually I use Excel to build it since I can customize and make it semi-automated manner.

Re: Importing a very large .txt file

JBerry — Mon, 25 Jan 2016 13:53:35 GMT

No such luck. : ( What started off as a favor has turned into a full blown project. Thanks for the pro tips!