Reading JSON files with SAS UNIX

[Areksoo]

Reading our JSON files works perfectly using the following code:

filename map temp;

libname blah json "&data_dir./JSONFILE.json" map=map automap=reuse;

However, we are trying to automate the process which means running in UNIX.  The above code only takes 10 minutes to run normally, but in UNIX, it takes 2 hours to run.

The JSON file is fairly complex where it will create 35 tables and when you look at the ALLDATA table, the variables go up to P8.  

Does anyone know what is causing this?  And also any solution to this problem?

[Tom]

Might just be a difference in the performance of the two machines.  Normally Unix machines are multi-user so it might be that your test ran at a time when the server had other heavy load.  Turn on the FULLSTIMER option and re-run the tests and see if the additional timing details in the log help you understand.

[Areksoo]

Unfortunately the issue is directly with the libname statement.  There's no timing stats returned in with a libname statement.

I've got a feeling that the cache is the issue as this process runs on the weekends where automated processes are at a minimum.  I have asked the testing team to use the option JSONCOMPRESS.  More CPU time, but less use of memory.  

[Tom]

In that case you might want to open a ticket with SAS Support so you can send them a copy of the JSON file so they can test.

If the file(s) uses a fixed definition you could make the LIBNAME statement run faster by not asking it to try to guess what the layout of the data is by examining the file. Instead create the map file yourself (perhaps you can just upload the one you created on Windows) and tell it to use that.

[Areksoo]

Unfortunately it's not fixed.  It's mostly fixed, but the development team will add new fields every so often... which is how it balloon to what it is now.   

[Tom]

---

@Areksoo wrote:

Unfortunately it's not fixed.  It's mostly fixed, but the development team will add new fields every so often... which is how it balloon to what it is now.   

---

Some programmers seem to think that using the flexibility of JSON (or XML) eliminates the need to plan out what your data fields are. Instead if just pushes the need onto the users of the data.