How to stop "data set limit reached" from appearing in my process flow.

[q5pham]

Hi Chris,

Thank you for your message. Is there any way that I could merge all the 5000 output tables in one along each ods output step, e.g merge table 1 and 2, then merge with table 3, etc. Even each of my single table is very small (3 rows and 6 columns), merging 5000 tables at once seems to be inefficient.

Thanks, Mai

[q5pham]

Hi Christ,

As most of my temp- datasets are in work folders, could you please show me the code that points to the Work folder instead of Arima folder as shown in your step 4:

libname arima "c:\projects\arimaout";

Thanks, Mai

[ChrisHemedinger]

Yes, here is a more complete example.  This works well if your output tables can all be named with a similar prefix -- for example, "AR".  This is a complete example that you should be able to run as-is to see how it works.

/* create a temp subfolder in WORK and assign a library to it */
options dlcreatedir;
%let outdir=%sysfunc(getoption(work));
/* temp library is ARIMA */
libname arima "&outdir./arima";


data nile;
   input level @@;
   year = intnx( 'year', '1jan1871'd, _n_-1 );
   format year year4.;
datalines;
1120  1160  963  1210  1160  1160  813  1230   1370  1140
995   935   1110 994   1020  960   1180 799    958   1140
1100  1210  1150 1250  1260  1220  1030 1100   774   840
874   694   940  833   701   916   692  1020   1050  969
831   726   456  824   702   1120  1100 832    764   821
768   845   864  862   698   845   744  796    1040  759
781   865   845  944   984   897   822  1010   771   676
649   846   812  742   801   1040  860  874    848   890
744   749   838  1050  918   986   797  923    975   815
1020  906   901  1170  912   746   919  718    714   740
;

data nile;
   set nile;
   AO1877 = ( year = '1jan1877'd );
   AO1913 = ( year = '1jan1913'd );
   LS1899 = ( year >= '1jan1899'd );
run;


/*-- ARIMA(0, 1, 1) Model --*/
proc arima data=nile ;
   identify var=level(1);
   estimate q=1 noint method=ml outmodel=arima.ar1;
   outlier maxnum= 5 id=year;
   
run;

/*-- MA1 Model with Outliers --*/
proc arima data=nile;
   identify var=level
            crosscorr=( AO1877 AO1913 LS1899 );
   estimate q=1
            input=( AO1877 AO1913 LS1899 )
            method=ml outmodel=arima.ar2;
   outlier maxnum=5 alpha=0.01 id=year;
run;

/* merge all similarly named output */
/* "ar:" means that all ARIMA.AR* data will be merged */
data WORK.arout;
  length ds $ 41 dsname $ 41;
  set arima.ar: indsname=ds;
  dsname=ds;
run;

/* Clear ARIMA lib so that EG doesn't detect it */
libname arima CLEAR;

[q5pham]

Wonderful! You really save me. Thank you so much Chris.