Exporting to Multiple PDF Files in a Loop by Appending

[apolitical]

Say I'm doing the following loop:

%do i=1 %to %eval(&count.);

    /*task 1, output from all loops goes into results_1.pdf*/

    /*task 2, output from all loops goes into results_2.pdf*/

    /*task 3, output from all loops goes into results_3.pdf*/

%end;

Tasks 1, 2 and 3 each produces some output I would like to save in 3 separate pdf files. Also, I want to save all outputs from task i into results_i.pdf, by appending results from every loop together. Looks like using ods pdf does not allow for easy appending, how can I do this? Thanks.

[Reeza]

Can you change the order of the loops to output all for task1 into a single PDF and then all of task2 into a single PDF?

---

@apolitical wrote:

Say I'm doing the following loop:

%do i=1 %to %eval(&count.);

    /*task 1, output from all loops goes into results_1.pdf*/

    /*task 2, output from all loops goes into results_2.pdf*/

    /*task 3, output from all loops goes into results_3.pdf*/

%end;

Tasks 1, 2 and 3 each produces some output I would like to save in 3 separate pdf files. Also, I want to save all outputs from task i into results_i.pdf, by appending results from every loop together. Looks like using ods pdf does not allow for easy appending, how can I do this? Thanks.

---

[apolitical]

Not sure what you mean by changing the order of the loop. One thing I can do is to break it up into 3 separate loops, and have an ods pdf file/close wrapping each of them, that could create what I want but needs a lot of repeated coding in my case. I'm curious if it can be done in one loop.

[Reeza]

---

@apolitical wrote:
Not sure what you mean by changing the order of the loop. One thing I can do is to break it up into 3 separate loops, and have an ods pdf file/close wrapping each of them, that could create what I want but needs a lot of repeated coding in my case. I'm curious if it can be done in one loop.

---

There shouldn't be any repeated code if you're just changing the loop orders. Usually doing it separately is more code but since this is a theoretical discussion it's difficult to say.

AFAIK, Once you close a PDF you can't append to it, except using a third party tool. 

The other option is using ODS DOCUMENT and GREPLAY but that's more work than reordering a loop for sure.

[apolitical]

Might be helpful to give a more concrete example, so you may show me where to place to ods commands.

%do i=1 %to %eval(&count.);

ods pdf file=".../results1.pdf"
    /*task 1, output from all loops goes into results_1.pdf*/
    proc reg data=dataset outest=param;
    model dep_var=var_&i.;
    output out=reg_out p=predicted r=residual 
    run;
ods pdf close;

ods pdf file=".../results2.pdf"
    /*task 2, output from all loops goes into results_2.pdf*/
    proc print data=param;run;
ods pdf close;

ods pdf file=".../results3.pdf"
    /*task 3, output from all loops goes into results_3.pdf*/
    proc gplot data=reg_out;
    plot predicted*time;
    run;
ods pdf close;

%end;

I want all regression outputs to be in pdf file 1, all print outputs in pdf file 2 and all plot outputs in pdf file 3. 

If I do it the way in the code above, only the last loop's results are saved in each of the files, because every iteration overwrites the previous one, instead of appending to the bottom of it. If I put ods pdf before the start of the loop and close it after the end of the loop, I can save everything I want, but it has to be in 1 big file.

Does that make my problem more clear? Thanks.

[Reeza]

Change your data structure - for the PROC REG transpose your input data and then add in a BY statement. You'll have one PROC REG and no macro needed.

https://blogs.sas.com/content/iml/2017/02/13/run-1000-regressions.html

---

@apolitical wrote:

Might be helpful to give a more concrete example, so you may show me where to place to ods commands.

%do i=1 %to %eval(&count.);

ods pdf file=".../results1.pdf"
    /*task 1, output from all loops goes into results_1.pdf*/
    proc reg data=dataset outest=param;
    model dep_var=var_&i.;
    output out=reg_out p=predicted r=residual 
    run;
ods pdf close;

ods pdf file=".../results2.pdf"
    /*task 2, output from all loops goes into results_2.pdf*/
    proc print data=param;run;
ods pdf close;

ods pdf file=".../results3.pdf"
    /*task 3, output from all loops goes into results_3.pdf*/
    proc gplot data=reg_out;
    plot predicted*time;
    run;
ods pdf close;

%end;

I want all regression outputs to be in pdf file 1, all print outputs in pdf file 2 and all plot outputs in pdf file 3. 

If I do it the way in the code above, only the last loop's results are saved in each of the files, because every iteration overwrites the previous one, instead of appending to the bottom of it. If I put ods pdf before the start of the loop and close it after the end of the loop, I can save everything I want, but it has to be in 1 big file.

Does that make my problem more clear? Thanks.

---

[apolitical]

Thanks. The real thing I'm trying to do is not quite like the example I gave. Besides, even if I use a by statement in proc reg, how can I separate the outputs into pdf files the way I want?

[Reeza]

---

@apolitical wrote:
Thanks. The real thing I'm trying to do is not quite like the example I gave. 

---

I can only comment on what you've posted and shown.

---

@apolitical wrote:
Besides, even if I use a by statement in proc reg, how can I separate the outputs into pdf files the way I want?

---

Try it. Use the demo from the link if you want, all the outputs go to where you want WITHOUT having any macro loops and your results are in the PDF files you want. One transpose and then the procs.

[ballardw]

To send all of the PROC reg outputs to the same PDF:

ods pdf file=".../results1.pdf"
    /*task 1, output from all loops goes into results_1.pdf*/
;/*  don't forget to end your ODS PDF statement!!!*/
%do i=1 %to %eval(&count.);
    proc reg data=dataset outest=param;
    model dep_var=var_&i.;
    output out=reg_out p=predicted r=residual 
    run;
%end;
ods pdf close;

This way the LOOP goes over the proc reg statements inside a single ODS PDF open/close pair.

Since your other two examples don't actually show any dependence on the Loop variable(and have incomplete ODS PDF statements)

I am not quite sure what you might want a loop for.

But if you want to group multiple PRINT of different outputs from PROC REG you need to make different output sets:

ods pdf file=".../results1.pdf"
    /*task 1, output from all loops goes into results_1.pdf*/
;/*  don't forget to end your ODS PDF statement!!!*/
%do i=1 %to %eval(&count.);
    proc reg data=dataset outest=param&i;
    model dep_var=var_&i.;
    output out=reg_out&i p=predicted r=residual 
    run;
%end;
ods pdf close;

And the printing all of the output sets would be something like:

ods pdf file=".../results2.pdf"
    /*task 2, output from all loops goes into results_2.pdf*/
;
%do i=1 %to %eval(&count.);

    proc print data=param&i;run;
%end;
ods pdf close;

And similar for the plot of the regression output REG_OUT sets. Though I would suggest moving to SGPLOT instead of GPLOT as the newer procedures are getting more development and features plus are designed to work with ODS output much better than the old device based procs like gplot.

Another thought would be having something that could use the value of the macro &I variable to make some meaningful titles...

[apolitical]

I think your edits are doing what I previously said can be done, basically break it apart into 3 separate macros, each accomplishing a task with numbered outputs from the regression in the first step.

But I'm looping through thousands of variables for experimental regressions, and that creates a large number of intermediate datasets for the system to handle. Therefore, I want to do everything in one macro and name generated datasets without &i, so the next iteration wipes clean the files generated in the previous iteration, all the while saving relevant information to separate external pdfs by appending.

[Reeza]

And transposing avoids all of that. No intermediate data sets, no macros, no loops.

---

@apolitical wrote:
I think your edits are doing what I previously said can be done, basically break it apart into 3 separate macros, each accomplishing a task with numbered outputs from the regression in the first step.

But I'm looping through thousands of variables for experimental regressions, and that creates a large number of intermediate datasets for the system to handle. Therefore, I want to do everything in one macro and name generated datasets without &i, so the next iteration wipes clean the files generated in the previous iteration, all the while saving relevant information to separate external pdfs by appending.

---

[Cynthia_sas]

Hi:

  I'm not entirely clear on what you want to do. Normally, ODS doesn't let you append to PDF files. However, you can have multiple PDF files open at the same time, as long as you use the ID= suboption to allow ODS to distinguish between what gets written to what output files.

  Here's an example:

ods pdf(id=all) file='c:\temp\all3.pdf';
 
ods pdf(id=one) file='c:\temp\just1.pdf';
proc print data=sashelp.class(obs=3);
  title '1) the first file';
run;
ods pdf(id=one) close;
 
ods pdf(id=two) file='c:\temp\just2.pdf';
proc print data=sashelp.shoes(obs=3);
  title '2) the second file';
run;
ods pdf(id=two) close;
 
ods pdf(id=tri) file='c:\temp\just3.pdf';
proc print data=sashelp.classfit(obs=3);
  title '3) the third file';
run;
ods pdf(id=tri) close;
ods pdf(id=all) close;

The first file c:\temp\all3.pdf will have all 3 report outputs because the destination is opened at the top of the 3 procedures with an ID= option and then each separate PROC step has a separate ODS PDF "sandwich" with a separate ID= option. Note how the ID=ALL instance of ODS PDF is not closed until the VERY VERY end of all the procedures.

It is a handy feature of ODS -- not sure it's what you want, but it might be.

Cynthia

asdfsadf

[Reeza]

/* 1. transpose from wide (Y, X1 ,...,X100) to long (varNum VarName Y Value) */
data Long;
set sashelp.heart;                       /* <== specify data set name HERE         */
array x [*] _numeric_;         /* <== specify explanatory variables HERE */
do varNum = 1 to dim(x);
   VarName = vname(x[varNum]);  /* variable name in char var */
   Value = x[varNum];           /* value for each variable for each obs */
   output;
end;
keep status varN: value;
run;

/* 2. Sort by BY-group variable */
proc sort data=Long;  by VarName;  run;

ods pdf file='c:\_localdata\temp\demo.pdf';

ods output parameterEstimates=P_Estimates;
proc logistic data=long;
by varName;
model status = value;
run;

ods pdf close;

[apolitical]

I made an executable example that can produce a pdf result to hopefully get across what I want to achieve. You only need to change the path definition in the following code and it should run fine.

This code runs 3 regressions, for each regression, 3 pages of outputs are saved. What I want is a code (the simple the better) that can generate 3 separate pdf files, with:

File number 1 combining page 1,4,7 of the product of this code; This is all the regression outcome.

File number 2 combining page 2,5,8 of the product of this code; This is all the key statistics.

File number 3 combining page 3,6,9 of the product of this code; This is all the plots.

data raw;
set sashelp.cars;
keep msrp invoice enginesize horsepower;
run;

proc contents data=raw out=cont;run;
proc sql;
create table count as 
select count(distinct name) as varcount
from cont
where lower(name)^="invoice"
;
quit;
data _null_;set count;call symputx ('count', varcount); run;
%put &count;

data varlist;set cont (where=(lower(name)^="invoice"));_num+1;run;

/*define pdf path here*/
%let path = ...;

%macro work;

ods pdf file="&path./allout.pdf";


	%do i=1 %to %eval(&count.);

		data _null_;set varlist;where _num=&i.;call symputx('var',name);run;
		%put &var.;

	/*task 1, records regression outcome*/
		ods output FitStatistics = fitstats;
		proc reg data=raw outest=param tableout;
		model invoice=&var. ;
		output out=reg_out p=predicted r=residual /*residual is actual - predicted values*/;
		run;

	/*task 2, gather model stats and output*/
		data stats1;
		set param;
		where _TYPE_="PARMS";
		keep &var. ;
		rename &var.=Estimate;
		run;

		data stats2;
		set param;
		where _TYPE_="PVALUE";
		if &var.<=0.1 then Sig_flag=1; else Sig_flag=0;
		keep &var. Sig_flag;
		rename &var.=P_value;
		run;

		data stats3;
		set fitstats;
		where label2="R-Square";
		keep nvalue2;
		rename nvalue2=R_squared;
		run;

		data stats4;
		set fitstats;
		where label2="Adj R-Sq";
		keep nvalue2;
		rename nvalue2=Adj_Rsquared;
		run;

		data stats_merge;
		retain iter Variable Estimate Sig_flag P_value R_squared Adj_Rsquared;
		merge stats1 stats2 stats3 stats4;
		iter=&i.;
		Variable="&var.";
		run;

		proc print data=stats_merge;run;

	/*task 3, plot observed and predcited values*/
		proc gplot data=reg_out;
		plot invoice*&var. predicted*&var. / overlay;
		run;

	%end;


ods pdf close;

%mend work;

%work;

[Cynthia_sas]

Hi: To rearrange and replay output, after it is created in one order, you might need to look at ODS DOCUMENT and PROC DOCUMENT. See this paper: https://support.sas.com/resources/papers/sgf09/318-2009.pdf and look at the "before" output on page 7, Figure 11 where the output is in "procedure-creation order" versus the "after" output on page 11, Figure 21 where the output is rearranged in Country order.

That might give you some other options for working with your output objects. Basically ODS DOCUMENT allows you to "freeze" the output objects in their original creation order after the procedures run and then later, you can come back and make a new folder structure in a new ODS DOCUMENT store and then you can copy and rearrange the original output objects in a new order. Once the objects are in a new order, you can replay all or some of them to an ODS destination in the new order.

Hope this helps,
Cynthia