BookmarkSubscribeRSS Feed
HeatherNewton
Quartz | Level 8
This Chris!
What is non-standard (user-defined) formats? my files are normal utf-8 or wlatin encoding sas datasets..

Can this output text file?

can this be run on server? as I have large amounts of very large files

Please assist. thanks.

HeatherNewton
Quartz | Level 8

I have installed and when I tried to open it it ask for login with password, how to use it?

HeatherNewton
Quartz | Level 8
Where do i go in powershell to execute the command u quoted above?
HeatherNewton
Quartz | Level 8
I managed to use with powershell but strangely no matter what size sas dataset i used i only get 3 lines in csv. How can i resolve this?
HeatherNewton
Quartz | Level 8
Works now! Thanks a lot! Chris, you are a super star!
HeatherNewton
Quartz | Level 8
On no seems all date not showing up in csv!
HeatherNewton
Quartz | Level 8
May be a bad sas dataset, i just tried with another seems date are time are fine
HeatherNewton
Quartz | Level 8

Hi Chris, one last question, I see that when I open the csv file with note, all entires are inside double quotes.. what is the difference between this csv and those csv without double quotes when open with note? 

I am worried as my main purpose is to convert sas dataset to text and later use in Oracle.. is this going to work for me too?

 

Thanks! 

ChrisHemedinger
Community Manager

That depends on how Oracle will ingest these CSV files. I think that PowerShell has a UseQuotes option on the Export-CSV cmdlet that might help. You could set this to "AsNeeded" and see if the result is better for you.

Register for SAS Innovate 2025!! The premier event for SAS users, May 6-9 in Orlando FL. Sign up now for the best deals!
HeatherNewton
Quartz | Level 8

thanks

 

Also is there a limit to how big a sas dataset is? If it exceed maximum no of rows excel can hold, then do I have to split the sas dataset into multiple sas data sets first?

 

 

 

 

Kurt_Bremser
Super User

@HeatherNewton wrote:

thanks

 

Also is there a limit to how big a sas dataset is? If it exceed maximum no of rows excel can hold, then do I have to split the sas dataset into multiple sas data sets first?

 


The size of a SAS dataset is, for all practical purposes, only limited by your available resources.

The limits of Excel play no role here. Only if you must create Excel spreadsheets, you need to first make your data fit into the 1M limit.

If your CSV is intended for a useful business intelligence application (which Excel is not, by parsecs), size will also not be an issue.

Sajid01
Meteorite | Level 14

Hello @HeatherNewton 
Yes,if the size of the dataset is large, then the dataset can be split into multiple files.
I see the question was first posted on 04/14/2022 and after over four months the issue is not yet resolved.
The optimal solution for your case is to use Python/R. (The best is however SAS).
Any one stool (R or Python depending on the comfort level) will solve all the requirements namely saving the data as text, csv, excel or uploading to Oracle and there would be no need to look around for multiple tools / solutions.
For large datasets the best practice would be save to an RDBMS system (Oracle, MySQL, PostgreSQL etc).

HeatherNewton
Quartz | Level 8

my biggest sas file is 1Terabyte, quite crazy.. is this still possible with python? 

 

how to split this with python, do you mean using read_sas in small chunk size?

then append the array and output to csv. I tried but strangely the csv is blank though no error messages. 

With read_sas, I ran into problem where there is missing values, I will get error message of 

Length of values does not match length of index

 

so I tried what Chris suggest using SAS local provider for OLE DB, seems no problem so far. but with SAS local provider method, I need to split sas data set into multiple data sets first as there is a limit to size of csv or text. I dont know how this could be done without SAS. Also this solution is for window only, I dont know if it could be used on server. 

 

If use Python, my plan is to have python installed on a server so not too limited by memory of pc. but I have not solved the problem with missing values in some column.

 

 

 

Sajid01
Meteorite | Level 14
Python does provide methods to handle large datasets.
The data can be read in chunks or number of rows.
Please have a look at this https://m.youtube.com/watch?v=l34l-90UF7U
By adjusting chunk size perhaps one may be able to make the process workable.
In addition there are libraries available and google search will be of help.
Now about 1 TB dataset /file I cannot say anything. I never had the need to handle such big datasets with python.





HeatherNewton
Quartz | Level 8

Chris, I want to ask you for SAS local provider for OLE DB, can I install it on a server with window? say if my server have window and 1 TB hard disk, does that mean I can install SAS local provider for OLE DB and convert sas data set of size 1TB to text, not csv? 

SAS Innovate 2025: Register Now

Registration is now open for SAS Innovate 2025 , our biggest and most exciting global event of the year! Join us in Orlando, FL, May 6-9.
Sign up by Dec. 31 to get the 2024 rate of just $495.
Register now!

How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

SAS Training: Just a Click Away

 Ready to level-up your skills? Choose your own adventure.

Browse our catalog!

Discussion stats
  • 56 replies
  • 5830 views
  • 6 likes
  • 12 in conversation