BookmarkSubscribeRSS Feed
ath_123
Calcite | Level 5

Hi,

 

I am having a requirement to create mutiple CSV files of 1GB size from a SAS Dataset . For the records of first 1GB data in my sas dataset, should go in my first file and the next set of 1 GB data should go in second file and so on...

Kindly let me know if more details required.

 

Any help on a solution to the requirement is highly appreciated.

 

 

 

2 REPLIES 2
Kurt_Bremser
Super User

You can use the FILEVAR= option in the FILE statement in your DATA _NULL_ step to define a variable that holds the name of the current file to be written.

 

Then RETAIN a counter variable that starts with 0 and is incremented in every iteration of the data step with the size of the current record (you need to calculate that by adding the lengths of your output variables; you could also cumulate the output line into a single variable and then use the length() function on that to determine the number of bytes you will add to that iteration; don't forget the line separator)

 

Once the counter reaches 1G, change the value of the FILEVAR variable and reset the counter to zero.

RW9
Diamond | Level 26 RW9
Diamond | Level 26

Thats a fairly large amount of data your dealing with then, if its multiples of 1gb in a CSV.  May I ask why you need to split it into multiple files.  A couple of other ideas:

Generate one CSV and stream it via HTTP

Generate one CSV and ZIP that using the inbuilt function of WinZip/RAR to split the archive into chunks of 1gb

Reassses if you really need to send that amount of data, what is the purpose of it, i.e. if they are just summarising it, why not summarise at your end and send the summarised data

Can you compress the data in any ways, for instance code longer text strings into small numbers, and provide a code list or formats catalog with the data - you would be amazed at the amount of storage space saved merely by changing a 20 character string into a 3 number coded value. 

sas-innovate-2024.png

 

Secure your spot at the must-attend AI and analytics event of 2024: SAS Innovate 2024! Get ready for a jam-packed agenda featuring workshops, super demos, breakout sessions, roundtables, inspiring keynotes and incredible networking events.

 

Register by March 1 to snag the Early Bird rate of just $695! Don't miss out on this exclusive offer. 

 

Register now!

What is Bayesian Analysis?

Learn the difference between classical and Bayesian statistical approaches and see a few PROC examples to perform Bayesian analysis in this video.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 2 replies
  • 810 views
  • 0 likes
  • 3 in conversation