topic Re: How to stop "data set limit reached" from appearing in my process flow. in SAS Enterprise Guide

How to stop "data set limit reached" from appearing in my process flow.

karim10 — Mon, 04 Nov 2013 16:56:23 GMT

Hi;

i am creating an project using SAS EG 4.3, and i want to limit the number of
tables that will appears in the process flow to the ones that i select( placed
in the process flow), so i named all my intermediate calculation table starting
by _TO_, which i think make them temporary and stop them from appearing in my
process flow, in the same time i need to keep some tables without showing them,
so I set the max number of data sets to appear to 0(zero). but now am stuck
with a warning note" data set limit reached" that appear in my
process flow.

so any one could please help me to find a way to stop this warning note from
appearing? Thanks.

Re: How to stop "data set limit reached" from appearing in my process flow.

MumSquared — Mon, 04 Nov 2013 23:37:49 GMT

Try

Menu tools / options - opens a window

Then on the left select Results General, on the right hand side make sure "Automatically add output data to the project tree" is unticked.

If you want to keep the datasets outside of the sas session declare a library, just remember to delete them when you no longer need them

Libname temp 'W:\SAS_Temp';

data temp.blahdata;

.....

Re: How to stop "data set limit reached" from appearing in my process flow.

ChrisHemedinger — Tue, 05 Nov 2013 13:38:34 GMT

Another trick is to allocate a library for your intended output data sets, and then "clear" the library before your program ends. The data set files remain in the folder, but as the allocated library is no longer defined EG won't add them to the project.

Then, in a separate step, you can reassign the library and add just the data sets you want.

Chris

Re: How to stop "data set limit reached" from appearing in my process flow.

Zachary — Tue, 14 Jul 2015 15:37:16 GMT

I unchecked the Automatically add button in my options and I still always get the Data Set Limit Reached.

I am using version 6.1 of Enterprise Guide.

Any suggestions on how to remove this for good?

Thank you.

Re: How to stop "data set limit reached" from appearing in my process flow.

SASKiwi — Tue, 14 Jul 2015 20:16:36 GMT

Try setting the option Maximum number of datasets added to project to 0. You will find this in the Options > Results > Results General.

This will mean no datasets are added to your process flow but they are still available in the Server List. If you still prefer them to be added increase the default number 50 to a limit that will not be reached.

Re: How to stop "data set limit reached" from appearing in my process flow.

Zachary — Tue, 14 Jul 2015 21:27:58 GMT

Still does not work. Below are the pictures of the checked and unchecked options, as well as the project tree.

Thank you.

Re: How to stop "data set limit reached" from appearing in my process flow.

SASKiwi — Tue, 14 Jul 2015 21:55:17 GMT

The message will only disappear if you set the default number to a limit bigger than the number of datasets you are producing (say 100, 200 etc.). Setting it to 0 will make the message appear all the time.

Re: How to stop "data set limit reached" from appearing in my process flow.

Zachary — Fri, 07 Aug 2015 20:42:05 GMT

This still does not work for me.

I am trying to upload my images, but I cannot do it in Communities for some reason right now.

Any thoughts without actually seeing my images? Maybe there is a way for me to post the files of the images.

Thank you.

Re: How to stop "data set limit reached" from appearing in my process flow.

SASKiwi — Fri, 07 Aug 2015 23:13:29 GMT

So have you tried my last suggestion of setting the number of datasets much higher, like 100, 200 etc?

Re: How to stop "data set limit reached" from appearing in my process flow.

Zachary — Mon, 10 Aug 2015 12:35:38 GMT

Yes, you are correct. The picture in the thread showed that (1) automatically adding the output to the tree is unchecked & (2) the maximum number of output files is 500.

Any other ideas are welcome. Thank you very much.

Re: How to stop "data set limit reached" from appearing in my process flow.

SASKiwi — Mon, 10 Aug 2015 19:46:19 GMT

How many datasets is your session producing? 500 as quite high, but you could push it up to 1000 or more.

Re: How to stop "data set limit reached" from appearing in my process flow.

Zachary — Mon, 10 Aug 2015 20:40:24 GMT

That's not the issue. I am only producing about half a dozen datasets.

Something must be getting lost in the translation, and I am probably poorly wording my issue. The second image posted above shows all of the datasets that are produced. It is slightly cleaner if I say to produce at most zero datasets. But then after each syntax module it has an arrow pointing to Data Set Limit Reached.

Perhaps this is just not doable as well.

Thank you.

Re: How to stop "data set limit reached" from appearing in my process flow.

ChrisHemedinger — Mon, 10 Aug 2015 20:57:55 GMT

Zach,

Is the "_TO_" trick working for you? That behavior was changed in 9.3:

http://blogs.sas.com/content/sasdummy/2011/03/29/undocumented-features-theres-a-reason-they-are-not-documented/

The only way that I know of to control which data sets appear in the process flow -- as output of a code node -- is to ensure:

- those data sets you want are created by the SAS program in the node -- or at least updated/touched by the program

- the data sets your don't want are either deleted before the end of the program, or at least reside in a library that can't be reached when EG calculates which data to add to the project. A trick would be to funnel non-essential data sets to a library that aliases to a different folder, then CLEAR the library definition at the end of the program. The files won't be deleted, but EG won't connect with them.

Chris

Re: How to stop "data set limit reached" from appearing in my process flow.

q5pham — Thu, 01 Oct 2015 06:16:24 GMT

Hi there,

My ods output from the proc arima should create more than 5000 tables, but the maximum I could get from EG is 500. Can anyone please help me to increase the maximum limit of output tables.

Thanks

Mai

Re: How to stop "data set limit reached" from appearing in my process flow.

ChrisHemedinger — Thu, 01 Oct 2015 11:57:34 GMT

I don't think you want 5000 tables to be added to your SAS Enterprise Guide project. That would make the project impossible to navigate (and very large and probably slow).

Here's what I suggest. Put all of this in your SAS program that runs the PROC ARIMA. It's important that all of these statements reside in the same program node, so they are run in one batch.

Create a special libname to store your PROC ARIMA output:

libname arima_t "c:\projects\arimaout";
Then following that, include the PROC ARIMA code (and whatever else you need for this analysis). Send the PROC ARIMA output to this ARIMA_T library (using OUT= or ODS OUTPUT if that's what you're using -- just make sure this library is referenced).
In the same program at the end, clear the ARIMA_T library (undefine it):

libname arima_t CLEAR;

This prevents EG from automatically adding data that was created using this library name.
Then re-assign a library of a different name to the same path:

libname arima "c:\projects\arimaout";

All of your output data will be present in the ARIMA library (in this example), but won't be added to the project (because EG won't detect the data sets automatically). You can use File->Open or use the Servers list to navigate the list of data sets in the new library.

Chris

Re: How to stop "data set limit reached" from appearing in my process flow.

q5pham — Mon, 05 Oct 2015 04:31:39 GMT

Hi Chris,

Thank you for your message. Is there any way that I could merge all the 5000 output tables in one along each ods output step, e.g merge table 1 and 2, then merge with table 3, etc. Even each of my single table is very small (3 rows and 6 columns), merging 5000 tables at once seems to be inefficient.

Thanks, Mai

Re: How to stop "data set limit reached" from appearing in my process flow.

q5pham — Mon, 05 Oct 2015 06:03:38 GMT

Hi Christ,

As most of my temp- datasets are in work folders, could you please show me the code that points to the Work folder instead of Arima folder as shown in your step 4:

libname arima "c:\projects\arimaout";

Thanks, Mai

Re: How to stop "data set limit reached" from appearing in my process flow.

ChrisHemedinger — Mon, 05 Oct 2015 12:52:14 GMT

Yes, here is a more complete example. This works well if your output tables can all be named with a similar prefix -- for example, "AR". This is a complete example that you should be able to run as-is to see how it works.

/* create a temp subfolder in WORK and assign a library to it */
options dlcreatedir;
%let outdir=%sysfunc(getoption(work));
/* temp library is ARIMA */
libname arima "&outdir./arima";


data nile;
   input level @@;
   year = intnx( 'year', '1jan1871'd, _n_-1 );
   format year year4.;
datalines;
1120  1160  963  1210  1160  1160  813  1230   1370  1140
995   935   1110 994   1020  960   1180 799    958   1140
1100  1210  1150 1250  1260  1220  1030 1100   774   840
874   694   940  833   701   916   692  1020   1050  969
831   726   456  824   702   1120  1100 832    764   821
768   845   864  862   698   845   744  796    1040  759
781   865   845  944   984   897   822  1010   771   676
649   846   812  742   801   1040  860  874    848   890
744   749   838  1050  918   986   797  923    975   815
1020  906   901  1170  912   746   919  718    714   740
;

data nile;
   set nile;
   AO1877 = ( year = '1jan1877'd );
   AO1913 = ( year = '1jan1913'd );
   LS1899 = ( year >= '1jan1899'd );
run;


/*-- ARIMA(0, 1, 1) Model --*/
proc arima data=nile ;
   identify var=level(1);
   estimate q=1 noint method=ml outmodel=arima.ar1;
   outlier maxnum= 5 id=year;
   
run;

/*-- MA1 Model with Outliers --*/
proc arima data=nile;
   identify var=level
            crosscorr=( AO1877 AO1913 LS1899 );
   estimate q=1
            input=( AO1877 AO1913 LS1899 )
            method=ml outmodel=arima.ar2;
   outlier maxnum=5 alpha=0.01 id=year;
run;

/* merge all similarly named output */
/* "ar:" means that all ARIMA.AR* data will be merged */
data WORK.arout;
  length ds $ 41 dsname $ 41;
  set arima.ar: indsname=ds;
  dsname=ds;
run;

/* Clear ARIMA lib so that EG doesn't detect it */
libname arima CLEAR;

Re: How to stop "data set limit reached" from appearing in my process flow.

q5pham — Mon, 05 Oct 2015 23:51:42 GMT

Wonderful! You really save me. Thank you so much Chris.