Import .txt data SAS on demand

[Tywop]

Hello guys, I am new here, first time using SAS. My name is Luciano and I am a brazilian student. 

I usually do my work with Stata software, however, this time, I needed to diversify my methods.

I want to import a governmental .txt file, but for some reason, it is impossible to read in Stata or R with normal functions as import delimited, read.delim, etc. So I opted for SAS because, for this data in this year specifically, they made inputs for reading the .txt file in SAS, which help the process if, of course, you have some expertise with SAS.

I am now using SAS on demand, but I am facing a lot of trouble trying to read the inputs. I will attach the input for the data. The error message that the SAS on demand is showing me is: 

It is really strange, because I write in the code just "C:\DADOS_ENEM_2006.TXT", without the "/pbr/biconfig/...".

Can anyone help me?

Sorry for the trouble. 

[Tom]

There is no C drive on the UNIX machine where the SAS code is running.

If you are using SAS on Demand then SAS is running on some server at SAS.  So first upload the file to your file space on that server and then use the name of the uploaded file in the code.  Not the name of where the file might live on the PC where you are running the browser that is connected to SAS on demand.

[Tywop]

Thanks for the quick response!

Hmmm, yes, I thought about that. But, how do I upload a file to my file space on the SAS Studio? Because I have just these options and the "upload" option is not working for some reason:

Is it even possible to upload a data file in SAS Studio? I am afraid of that because I am using the free option of SAS.

Thanks for your patience.

[Tom]

The GUI is clunky.

Try clicking on your HOME folder.  It seems to be displayed as Files (Home) in that photograph.

Then the UPLOAD button should be active.

[Tywop]

Yeap, it started working again.
Unfortunately, the limit for upload is only 1GB and my text file is 2.7 GB.
There is other way to make this work? I tried everything, and this is my last option. I just need to read the .txt file via the input .sas file and them transfer the data to .dta (for Stata software).
Sorry for the despair and thanks again for the patience.

[Reeza · Accepted solution]

You can upload the file to a cloud storage and read it via a URL. However, once you read it, the file will likely get bigger and you're going to have trouble processing that with SAS On Demand as the files will be too large. If you work with a University you'll want to get a software version to install on your computer directly or on a cloud platform somewhere. 

SAS On Demand isn't designed for large scale processing of data.

[Tom]

---

@Tywop wrote:
Yeap, it started working again.
Unfortunately, the limit for upload is only 1GB and my text file is 2.7 GB.
There is other way to make this work? I tried everything, and this is my last option. I just need to read the .txt file via the input .sas file and them transfer the data to .dta (for Stata software).
Sorry for the despair and thanks again for the patience.

---

ZIP the file. Text files usually compress down to < 20 % of the original file size.

You can use the ZIP fileref engine when reading the file.

But if you have STATA then why not just write the STATA code to read the text file?

[Tywop]

Thanks for the support Tom.

The thing is, for I don't know reason, this .txt file use delimiters that Stata or R cannot read easily. I tried a lot, I really did, but I couldn't write a Stata code to read the text file. That is why I am using the SAS as my last option.
Tom, if it won't be asking too much, could you write the code with that "ZIP fileref engine" that you mentioned? I have, sinceraly, no idea how to write anything in SAS, I am just using the software to get this specifically data.
Again, thanks for everything.

[Reeza]

You would need to provide file layout.

[Tywop]

Thank you for the answer.
Sorry, but what are you referring to a "file layout"?
The only thing I can provide is the input that was already made for this data. It is attached in the first message in this thread, but I can attach it again.

This is the "layout" that you mentioned? If it is not, could you help me by describing it a little more please?

Thanks for the patience.  

[Reeza]

DATA ENEM_2006;
RETAIN MASC_INSCRITO NU_ANO DT_NASCIMENTO;
INFILE "C:\DADOS_ENEM_2006.TXT" LRECL=696 MISSOVER;

becomes this:

*reference to zip file;
filename DADOS ZIP '/home/userName/testzip.zip' member=DADOS_ENEM_2006.TXT;

DATA ENEM_2006;
RETAIN MASC_INSCRITO NU_ANO DT_NASCIMENTO;
INFILE DADOS LRECL=696 MISSOVER;

*rest of your code;

https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.2/lestmtsglobal/n1dn0f61yfyzton1l2ngsa1clllr.ht...

https://kb.iu.edu/d/azva#:~:text=You%20can%20read%20such%20a,newdata%3B%20INFILE%20ZIPFILE(dataset.

However, I suspect you're still going to run into the same issue at the end of the day, the file is to big. Is your goal to write it out to a different format to pipe it back to Stata or R?

FYI - this is a fixed width file, there is no delimiter because the position denotes the column/variable, you can use R to read such a file but it's work to specify the locations which the SAS code already does.

https://stat.ethz.ch/R-manual/R-devel/library/utils/html/read.fwf.html

[Tom]

Only delimited files (like a CSV file) have delimiters.

Just read the file as a normal file.

https://www.stata.com/support/faqs/data-management/reading-fixed-format-data/