Exporting large data set to bookmarked PDF

[elwayfan446]

Good morning...

I have a very large data set with over 17,000,000 records that I am trying to export to a bookmarked PDF.  I would like to index and catalog that PDF so my users can do a search by loan for those records.  Each time I have tried to export that via Enterprise Guide it runs for over 24 hours and then fails on a memory or not enough space error.  I don't have an easy way to break it up into smaller files by loan number unless someone can make a suggestion on that as well.

Any ideas would be appreciated!

[Reeza]

Does your data have some ideal way to break it up? Ie some grouping system? There is a NEWFILE option in ODS PDF that you can use to separate the files automatically.

http://support.sas.com/documentation/cdl/en/odsug/61723/HTML/default/viewer.htm#a002217095.htm#a0029...

[elwayfan446]

I had grouped it by Account Number and designated Enterprise Guide to start each group on a new page. When I tested this with the top 10 records, it seemed to work fine.  The first account was grouped with its observations expanded below it.  I can't really group it in SAS because each observation has the account number tied to it with data that is not unique.

Do you think it is better to add a program with SAS code to do this?  Would you have an example you can show me?

[SASKiwi]

Personally I don't think a single PDF will scale to 17M records.

I suggest you try experimenting with 1,000, 10,000, 100,000, 1M record PDFs first. Can you produce them? Can you open them and search for a loan in a reasonable timeframe?

I've found large ODS reports use a lot processing resources so I'm not surprised you are running into resource issues.

You really should consider a database-type solution for the volume of data you have.

[ballardw]

You may also want to close the HTML (and/or LISTING) destination while doing such if you haven't already. Trying to build both the PDF and HTML tables could be using LOTS of resources.

[elwayfan446]

Thank you @SASKiwi and @ballardw...

@SASKiwi... I have not tried to break them down yet. I was trying to figure a way to easily split the datasets and then print individual pdfs so I am still looking at that.

@ballardw, I did ensure all output was off except for PDF but that was a great idea. The first few times I tested I realized I had that on.