Reading many XML files - complex and hierarchical - performance tests available???

[kbohle]

Hello,

we have to read many (roundabout 100.000 / day) XML files into SAS format.

Does someone have any experience / performance testing done to read a mass of XML files at once?

What seems to be the best method? (Libname, Proc XSL, own text parser, ...)

The XML files are complex and hierarchical and have a size between 10 KB and 4 MB and all in all three different formats of files.

Thanks in advance.    

[BillM_SAS]

The XML LIBNAME engine is probably the best place to start. Since you state that the XML files are complex, you will most likely need the XML Mapper. This application will create a XMLMap file to allow the XML LIBNAME engine to read the hierarchical XML files into rectangular SAS data sets.

This XSL procedure transforms a XML file to a different XML structure or another file format like text or HTML. Unless the XML is too complex for the XML Mapper, the XSL procedure will most likely not be needed.

[kbohle]

Thanks for your experience.

Now - one step further - I tested already the libname engine and get a lot of problems.

The files are too complex and large for the libname engine - I've got a MEMSIZE problem. SAS support told me that I have to change MEMSIZE to 8GB which is currently in the used environment not possible.

PEOC XSL is working well to transform some of files into CSV format. Do you have any configuration experiences with PROC XSL?

Thanks in advance.

[kbohle · Accepted solution]

For all who are interested in XML computation with SAS.

I found know a solution together with the SAS support.

I was able to transform roundabout 2500 XML files in one XSLT computation into one CSV-file. I used within SAS 9.2 the PROC XSL procedure. In addition I had to change the SAS config file (JREOPTION) to increase the Memory size used for Java.

I had to transform three kinds of XML files (each 2500 files - together 7500 files)with different structure, different complexity and different sizes.

The transformation of the first type - in total 55 MB size - was transformed within 18 seconds!!!

The transformation of the second type - in total 145 MB size - was transformed within 60 seconds!!!

The transformation of the first type - in total 34 MB size - was transformed within 10 seconds!!!

Kind regards