topic Re: File Size Limitation in XML Mapper in SAS Programming

File Size Limitation in XML Mapper

Babloo — Wed, 17 Jan 2018 06:29:13 GMT

When I read a 55K records from XML file via SAS XML Mapper to create a .map file, it prompts an error message "The file size exceeds the recommended maximum. ......".

Can Mapper read only small XML files < 5K or less?

Re: File Size Limitation in XML Mapper

BrunoMueller — Wed, 17 Jan 2018 07:42:34 GMT

Have a look at this Usage note http://support.sas.com/kb/39/804.html

It explains various ways how to handle this situation.

I usually recommend to make the XML file smaller and build the map file.

Re: File Size Limitation in XML Mapper

Babloo — Wed, 17 Jan 2018 09:16:10 GMT

Could you please help me understand how can I achieve the following?

Cut the XML file where the elements repeat and map this file, which is a subset of the larger XML file.
If you have the schema, create an XMLMap file from the schema rather than the XML file.

Re: File Size Limitation in XML Mapper

BrunoMueller — Wed, 17 Jan 2018 10:21:26 GMT

for 1

Assume the following XML

<?xml version="1.0" encoding="iso-8859-1" ?>
<NHL>
  <CONFERENCE> Eastern 
    <DIVISION> Southeast  
      <TEAM name="Thrashers"  abbrev="ATL" />  
      <TEAM name="Hurricanes" abbrev="CAR" />  
      <TEAM name="Panthers"   abbrev="FLA" />  
      <TEAM name="Lightning"  abbrev="TB" />  
      <TEAM name="Capitals"   abbrev="WSH" />  
   </DIVISION>
 </CONFERENCE> 

 <CONFERENCE> Western
   <DIVISION> Pacific  
     <TEAM name="Stars"   abbrev="DAL" />  
     <TEAM name="Kings"   abbrev="LA" />  
     <TEAM name="Ducks"   abbrev="ANA" />  
     <TEAM name="Coyotes" abbrev="PHX" />  
     <TEAM name="Sharks"  abbrev="SJ" />  
   </DIVISION>
  </CONFERENCE> 
</NHL>

<TEAM .../> is the repeating element that will make up one observation

we only need one <TEAM ... /> entry to build the XML map file, the others can be deleted.

Also <CONFERENCE> and <DIVISION> appear more than once, but we only need them once

For 2

If you have a XSD you can build the map file from this

File -> Open XML Schema

Tools -> Automap using XSD

the Map file created might not be what you want

Re: File Size Limitation in XML Mapper

BrunoMueller — Wed, 17 Jan 2018 10:24:11 GMT

Have a go at creating the map file automatically, see the doc for more information

Importing an XML Document Using the AUTOMAP= Option to Generate an XMLMap

Re: File Size Limitation in XML Mapper

Babloo — Wed, 17 Jan 2018 10:28:31 GMT

Per the document ,the following SAS statements import the XML document Nhl.xml. In my case, I do not have .map file to get it done. Could you please guide me how to proceed now?

filename nhl 'C:\Example\Nhl.xml'; /*1*/
filename map 'C:\Output\NhlGenerate.map'; /*2*/

libname nhl xmlv2 automap=replace xmlmap=map; /*3*/

proc print data=nhl.team; /*4*/
run;

Re: File Size Limitation in XML Mapper

Babloo — Wed, 17 Jan 2018 10:33:13 GMT

I'm trying to understand the first part of your response. In the example below, although <TEAM .../> is the repeating element, each of the element seem to have different record and I'm not certain how can I make it to one observation. Appreciate if you could guide me here.

Re: File Size Limitation in XML Mapper

BrunoMueller — Wed, 17 Jan 2018 10:34:38 GMT

there is no way around reading the documentation!

check the description for /*3*/

Re: File Size Limitation in XML Mapper

Babloo — Wed, 17 Jan 2018 10:54:44 GMT

Whether the 'team' refers to a SAS dataset below?

proc print data=nhl.team; /*4*/
run;

Re: File Size Limitation in XML Mapper

Babloo — Wed, 17 Jan 2018 13:31:39 GMT

I tried the following code, but I'm not finding the .map file in /wrk1. There is no error or warning in the log file. Appreciate if you could guide me here to generate .map file.

LIBNAME test "/shared/sas" ;
filename testing "/wrk1/dynamic_new.xml"; /*1*/
filename map "/wrk1/dynamic_new.map"; /*2*/

libname testing xml xmlmap=map; /*3*/

data dynamic; 
set test.DYNAMIC(obs=1000); 
run;

Re: File Size Limitation in XML Mapper

BrunoMueller — Wed, 17 Jan 2018 13:53:43 GMT

Which SAS version are you using?

Run this code and provide the log

%put NOTE: &=sysscpl;
%put NOTE: &=sysvlong;

Your example code differs quite a bit from the example in the doc:

for instance your are missing the automap=replace

you are also using the XML versus the XMLV2 engine

Any reasons for the changes?

Run Proc CONTENTS on the testing library to see which tables are available, then you can use a DATA Step to read the actual data.

Re: File Size Limitation in XML Mapper

Babloo — Wed, 17 Jan 2018 14:12:20 GMT

Here you go.

23         GOPTIONS ACCESSIBLE;
24         %put NOTE: &=sysscpl;
NOTE: &=sysscpl
25         %put NOTE: &=sysvlong;
NOTE: &=sysvlong

I'm using SAS 9.2 and if I use automap=replace and XMLV2 I used to receive an error.

33         proc contents data=testing._all_ nods;
34         run;

ERROR: Physical file does not exist, /wrk1/dynamic_new.map.

Re: File Size Limitation in XML Mapper

BrunoMueller — Wed, 17 Jan 2018 14:26:26 GMT

So I see, the AUTOMAP= option is available beginning with SAS® 9.3 TS1M2, so it is not available to you.

You have to go to the approach of reducing the size of your XML file as explained earlier and then build the map from it.

If you know what you want to read from the XML files you can build the map file yourself with an text editor.

Some editors, like Notepad++, have support to actually look at the structure, you may also use your browser to look at the content of the xml file to figure out the structure.

On linux system you might have the following command

xmllint –format file.xml

available to make an XML easier to read

Re: File Size Limitation in XML Mapper

Babloo — Wed, 17 Jan 2018 14:51:17 GMT

Whether It is easy to reduce the size of my XML file when it is generated from the SAS dataset which has 55000 records and 25 variables to build the map from it?

Re: File Size Limitation in XML Mapper

BrunoMueller — Wed, 17 Jan 2018 15:19:58 GMT

I do not understand, if your XML file is created from a SAS data set, why would you want to read it again into SAS?

Since your XML file is created from a SAS data set, you can control the number of obs written to that file, so you can build your XML map file using the SAS XML Mapper program.

Re: File Size Limitation in XML Mapper

Babloo — Wed, 17 Jan 2018 15:25:35 GMT

Could you please help me to know how can I create the Map file from SAS XML
mapper?

Re: File Size Limitation in XML Mapper

BrunoMueller — Wed, 17 Jan 2018 15:30:58 GMT

Have a look here
https://blogs.sas.com/content/sgf/2016/06/24/tips-for-reading-xml-files-into-sas-software/

Re: File Size Limitation in XML Mapper

KevinViel — Thu, 18 Jan 2018 00:15:42 GMT

Being guilty of very, very briefly perusing the link, I wrote my own map file for the CDISC define.xml (perhaps no substitute for the SAS Clinical Standards Toolkit). I spent a weekend morning over a prolonged breakfast and reading https://www.w3schools.com/xml/. With a nod to Jozef Aerts (XML4Pharma) on the Linkedin CDISC Define-XML group, I learned the hard way about the importance of namespaces in XML.

With a few references in hand, I can write XML (or write a SAS program that writes XML "manually"). The OP did not specify the nature of the XML, but very likely a group exists that will provide useful advice, or, in my usual case, corrections.

HTH,

Kevin

Re: File Size Limitation in XML Mapper

Babloo — Thu, 18 Jan 2018 05:24:33 GMT

<?xml version="1.0" encoding="iso-8859-1" ?>
<NHL>
  <CONFERENCE> Eastern 
    <DIVISION> Southeast  
      <TEAM name="Thrashers"  abbrev="ATL" />  
      <TEAM name="Hurricanes" abbrev="CAR" />  
      <TEAM name="Panthers"   abbrev="FLA" />  
      <TEAM name="Lightning"  abbrev="TB" />  
      <TEAM name="Capitals"   abbrev="WSH" />  
   </DIVISION>
 </CONFERENCE> 

 <CONFERENCE> Western
   <DIVISION> Pacific  
     <TEAM name="Stars"   abbrev="DAL" />  
     <TEAM name="Kings"   abbrev="LA" />  
     <TEAM name="Ducks"   abbrev="ANA" />  
     <TEAM name="Coyotes" abbrev="PHX" />  
     <TEAM name="Sharks"  abbrev="SJ" />  
   </DIVISION>
  </CONFERENCE> 
</NHL>

In the above example, could you please help me understand how will you cut the repeating elements?

Re: File Size Limitation in XML Mapper

BrunoMueller — Thu, 18 Jan 2018 07:58:30 GMT

Like so

<?xml version="1.0" encoding="iso-8859-1" ?>
<NHL>
  <CONFERENCE> Eastern 
    <DIVISION> Southeast  
      <TEAM name="Thrashers"  abbrev="ATL" />  
   </DIVISION>
 </CONFERENCE> 
</NHL>