topic Re: Reading and extracting data from large files in SAS Programming

Reading and extracting data from large files

Jacob1602 — Wed, 27 Apr 2016 06:58:32 GMT

I have a file that are extreamly large. Larger the 32.000.

I need SAS to read it, and the way i want it to do so is, that I expect to infile the first 32.000 bytes fine my tang <tag01> Then I will extract a new subset of the file starting with <tag01> and extracting to <tag02>, thereafter extract form <tag02> to <tag03> and so forth.

Thereafter I will drop som of the extracted segments and rebuild the file again. What I will leave out is some databloks, that are extreamly large, and when I rebuild the file the size is less the 32000.

Re: Reading and extracting data from large files

Ksharp — Wed, 27 Apr 2016 07:12:10 GMT

It looks like it is a XML file . Try XMLV2 engine.

libname x xmlv2 '/folders/myfolders/xx.xml' automap=replace xmlmap='/folders/myfolders/xx.map';

Re: Reading and extracting data from large files

RW9 — Wed, 27 Apr 2016 08:28:43 GMT

As @Ksharp has stated it sounds like you are working with XML files. I would not recommend you try to read and modify these yourself, you can use the libname as @Ksharp has provided, and for more complex XML then the XML Mapper. However to properly work with XML data you would be far better off using software built to work with XML. There are plenty of paid/free ones out there:

https://en.wikipedia.org/wiki/Comparison_of_XML_editors

XMLFox springs to mind. These allow you to alter/query/update XML files as if they were a database, i.e. searhing for tags, removing blocks etc.

Re: Reading and extracting data from large files

Jacob1602 — Wed, 27 Apr 2016 09:59:03 GMT

Thanks for the advise, and you are totally right, it is a xml file. The problem is, that the file is to big for the SAS xml engine due to very big tatabloks. Therefore I will recuce the xml file by extracting the databloks and inserting a surogat-key instead. Thereby, when I read the xml file, I have a perfect dimensional model, with the databolks as my fact and the other datasets from the xml as my dimensions.

We are an organisation that is very SAS centric, therefore I will prefere to stick to SAS code.

Re: Reading and extracting data from large files

Ksharp — Thu, 28 Apr 2016 00:58:41 GMT

It is NOT good for hard coding . If your XML is quite simple, you don't need too much code. But it is complicated ,I am afraid that would be very hard for SAS coding .