BookmarkSubscribeRSS Feed
☑ This topic is solved. Need further help from the community? Please sign in and ask a new question.
A_SAS_Man
Pyrite | Level 9

I am using the json engine to parse some very large datafiles. I notice in the process of parsing these files that it creates a dataset of "ALLDATA" which seems to just be some sort of appending of all the other tables/objects that isn't super useful to me. From the reading I've done my understanding is this is an artifact of how sas processes json files rather than how a json file is structured necessarily (but open to corrections there). Is there a way to have it skip doing this to help speed up the process? Code I'm using is very simple but see below.

 

filename dat ZIP '/data/file.json.gz' GZIP;

libname dat json;
1 ACCEPTED SOLUTION

Accepted Solutions
Tom
Super User Tom
Super User

@A_SAS_Man wrote:

I am using the json engine to parse some very large datafiles. I notice in the process of parsing these files that it creates a dataset of "ALLDATA" which seems to just be some sort of appending of all the other tables/objects that isn't super useful to me. From the reading I've done my understanding is this is an artifact of how sas processes json files rather than how a json file is structured necessarily (but open to corrections there). Is there a way to have it skip doing this to help speed up the process? Code I'm using is very simple but see below.

 

filename dat ZIP '/data/file.json.gz' GZIP;

libname dat json;

That code is not going to create any ALLDATA dataset, it is just defining the LIBREF that points to the JSON file.  

If you don't want to read DAT.ALLDATA then don't read it.

 

But there is an option that should suppress it.  Try it and see if it works in your version.

https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/lestmtsglobal/n1jfdetszx99ban1rl4zll6tej7j.ht...

NOALLDATA

Suppresses creation of the ALLDATA data set.

Note This option is available beginning with SAS Viya 3.4.

 

 

View solution in original post

2 REPLIES 2
Tom
Super User Tom
Super User

@A_SAS_Man wrote:

I am using the json engine to parse some very large datafiles. I notice in the process of parsing these files that it creates a dataset of "ALLDATA" which seems to just be some sort of appending of all the other tables/objects that isn't super useful to me. From the reading I've done my understanding is this is an artifact of how sas processes json files rather than how a json file is structured necessarily (but open to corrections there). Is there a way to have it skip doing this to help speed up the process? Code I'm using is very simple but see below.

 

filename dat ZIP '/data/file.json.gz' GZIP;

libname dat json;

That code is not going to create any ALLDATA dataset, it is just defining the LIBREF that points to the JSON file.  

If you don't want to read DAT.ALLDATA then don't read it.

 

But there is an option that should suppress it.  Try it and see if it works in your version.

https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/lestmtsglobal/n1jfdetszx99ban1rl4zll6tej7j.ht...

NOALLDATA

Suppresses creation of the ALLDATA data set.

Note This option is available beginning with SAS Viya 3.4.

 

 

A_SAS_Man
Pyrite | Level 9

When I navigate to the "dat" library I can see an ALLDATA file. It looks like I may have too old of a version of SAS to use that option with unfortunately, I'm just using SAS EG I'm not familiar with Viya.

 

Edit: It seems as though the NOALLDATA keyword might be working for me even though it wasn't showing up before... Testing it out now and will accept this if it works.

sas-innovate-2024.png

Available on demand!

Missed SAS Innovate Las Vegas? Watch all the action for free! View the keynotes, general sessions and 22 breakouts on demand.

 

Register now!

How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 2 replies
  • 389 views
  • 0 likes
  • 2 in conversation