topic Re: Exporting large data set to bookmarked PDF in SAS Enterprise Guide

Exporting large data set to bookmarked PDF

elwayfan446 — Thu, 03 Dec 2015 15:52:02 GMT

Good morning...

I have a very large data set with over 17,000,000 records that I am trying to export to a bookmarked PDF. I would like to index and catalog that PDF so my users can do a search by loan for those records. Each time I have tried to export that via Enterprise Guide it runs for over 24 hours and then fails on a memory or not enough space error. I don't have an easy way to break it up into smaller files by loan number unless someone can make a suggestion on that as well.

Any ideas would be appreciated!

Re: Exporting large data set to bookmarked PDF

Reeza — Thu, 03 Dec 2015 16:02:00 GMT

Does your data have some ideal way to break it up? Ie some grouping system? There is a NEWFILE option in ODS PDF that you can use to separate the files automatically.

http://support.sas.com/documentation/cdl/en/odsug/61723/HTML/default/viewer.htm#a002217095.htm#a002985640

Re: Exporting large data set to bookmarked PDF

elwayfan446 — Thu, 03 Dec 2015 16:55:19 GMT

I had grouped it by Account Number and designated Enterprise Guide to start each group on a new page. When I tested this with the top 10 records, it seemed to work fine. The first account was grouped with its observations expanded below it. I can't really group it in SAS because each observation has the account number tied to it with data that is not unique.

Do you think it is better to add a program with SAS code to do this? Would you have an example you can show me?

Re: Exporting large data set to bookmarked PDF

SASKiwi — Thu, 03 Dec 2015 23:24:04 GMT

Personally I don't think a single PDF will scale to 17M records.

I suggest you try experimenting with 1,000, 10,000, 100,000, 1M record PDFs first. Can you produce them? Can you open them and search for a loan in a reasonable timeframe?

I've found large ODS reports use a lot processing resources so I'm not surprised you are running into resource issues.

You really should consider a database-type solution for the volume of data you have.

Re: Exporting large data set to bookmarked PDF

ballardw — Thu, 03 Dec 2015 23:39:49 GMT

You may also want to close the HTML (and/or LISTING) destination while doing such if you haven't already. Trying to build both the PDF and HTML tables could be using LOTS of resources.

Re: Exporting large data set to bookmarked PDF

elwayfan446 — Fri, 04 Dec 2015 00:34:34 GMT

Thank you @SASKiwi and @ballardw...

@SASKiwi... I have not tried to break them down yet. I was trying to figure a way to easily split the datasets and then print individual pdfs so I am still looking at that.

@ballardw, I did ensure all output was off except for PDF but that was a great idea. The first few times I tested I realized I had that on.