Data size limitations for running code

[J111]

Hello,

Are there data size limitations for the following code,

if so is there a solution rather than a smaller data input:

----------------------------------------------------

filename sas_code temp ;

data _null_ ; 

     Set MATCHING_TABLE_FUNC end = last ;

     file sas_code ; 

if _N_ = 1 then put "options ls = 256 errors = max nospool ; Data Formula_ ; set TRNSP_SERIES_LIST ;" ; 

     put Target '='  Formula ";" ;

if last then put "Run ;" ; 

run ; 

data _null_ ;

     infile sas_code ;

     input ;

     put _infile_ ;

run ;

%include sas_code ;

----------------------

The log has an error of connection to the server was lost.

The matching table includes thousands of formula such as

target= x1 + x2

The   TRNSP_SERIES_LIST SAS file includes 487 rows and 4624 columns.

If the  TRNSP_SERIES_LIST includes only 175 rows there is no log error of connection and the program runs smoothly.

We work with Linux, SAS EG 8.2

-------------------------

I can invent data if it is required.

Thanks in advance.

[Ksharp]

Can you post dataset "TRNSP_SERIES_LIST " ,especially variable "Target" and "Formula" .

[J111]

Hello,

The TRNP_SERIES_LIST columns looks like this

DATE x1 x2 x3  until x4623

The x(i) are variables from TRNS_SERIES_LIST that appear in the formula in the MATCHING_TABLE.

The matching table includes target name of a new variable to calculate and the formula of variables.

for example new_var = x1 + x2 /x3 + x4

[Kurt_Bremser]

How many observations does matching_table_func have?

How are variables target and formula defined?

I would run this code with dataset option OBS=1 in the first step, to check the basic validity. From there, expand the OBS number.

[J111]

Hello,

I will try to answer your questions:-

The code is valid it runs very good for 175 rows in the TRNS_SERIES_LIST table.

The matching table includes 2454 rows each row with one target formula.

Each formula includes up to 20 variables.

The TRNS_SERIES_LIST includes one column of date and thousands columns of variables.

These variables appear in  the formula.

Thanks

[J111]

Target and formula are character variables in the matching table.

[J111]

Hello,

The following log error

Error: The connection to the server has been lost.

appears for (obs=177)

for obs with lower values such as 170, 175, 176 the program runs fine.

------------------------------------------------------------------------------------

filename sas_code temp ;

data _null_ ; 

     Set MATCHING_TABLE_FUNC end = last ;

     file sas_code ; 

if _N_ = 1 then put "options ls = 256 errors = max nospool ; Data Formula_ ; set TRNSP_SERIES_LIST(obs=177) ;" ; 

     put Target '='  Formula ";" ;

if last then put "Run ;" ; 

run ; 

data _null_ ;

     infile sas_code ;

     input ;

     put _infile_ ;

run ;

%include sas_code ;

[ballardw]

---

@J111 wrote:

Hello,

The following log error

Error: The connection to the server has been lost.

appears for (obs=177)

for obs with lower values such as 170, 175, 176 the program runs fine.

Does it ALWAYS fail at OBS=177?

What is the size of the text file written?

If it repeatedly fails at the same OBS number then you may be running into a local file space limit, at in user files at that location are limited to XXXX bytes. That would indicate time to ask your admin.

If always fails at 177 have you looked at the content of OBS=177?, perhaps the values has something in it causing a problem. Make a version of the data set that drops the current observation 177 and see if that runs.

If it occurs at different observations you likely have some other network interference and should talk to your IT support.

You may also need to describe your SAS environment. 

[Kurt_Bremser]

Restrict the obs of the formula table:

data _null_ ; 
     Set MATCHING_TABLE_FUNC (obs=1) end = last ;
     file sas_code ; 
if _N_ = 1 then put "Data Formula_ ; set TRNSP_SERIES_LIST;" ; 
     put Target '='  Formula ";" ;
if last then put "Run ;" ; 
run ; 

and see if it still fails at obs 176 of trnsp_series_list.

[J111]

Hello,

I am running several combinations of obs=

for the combination below, the program runs fine.

once increase obs from 300 to 350 the connection to the server is lost.

I shall check all suggestions posted and update regarding the results.

------------------------

filename sas_code temp ;

data _null_ ; 

     Set MATCHING_TABLE_FUNC(obs=2500) end = last ;

     file sas_code ; 

if _N_ = 1 then put "options ls = 256 errors = max nospool ; Data Formula_ ; set TRNSP_SERIES_LIST(obs=300) ;" ; 

     put Target '='  Formula ";" ;

if last then put "Run ;" ; 

run ; 

data _null_ ;

     infile sas_code ;

     input ;

     put _infile_ ;

run ;

%include sas_code ;

[Kurt_Bremser]

Run this:

data _null_;
set MATCHING_TABLE_FUNC end=last;
file sas_code;
if _N_ = 1 then put "options ls = 256 errors = max nospool; Data Formula_; set TRNSP_SERIES_LIST(obs=10);"  
put Target '='  Formula ";";
if last then put "run;";
run;

%include sas_code;

and then inspect the log for possible problems with the created code.

Since you add several thousand new variables (each with 8 bytes if numeric), if the source dataset already has a large observation size you might run against a limit in your WORK.

[J111]

Hello,

I could not find code issues with obs=10

the only note is

NOTE: Mathematical operations could not be performed at the following places. The results of the operations have been set to missing values

-----

We have at least 150 GIGA space for work on the server.

----

Matching table has less than 2500 rows.

Why by limiting it to (obs=2500) the trns_table could be larger than (obs=177) ? and even (obs=300)

---

there is no consistency - next run with combination of 2500 & 300 ended with connection error.

Thanks

[Kurt_Bremser]

Any diligent SAS admin will establish usage quotas on servers, to prevent one user/program from eating up everything. So your individual quota may be much smaller than the overall size of the storage volume.

But the fact that this does not happen consistently lets me suspect outside interference (network issues, router timeouts, anti-malware software etc).

Run the same identical code multiple times in succession, and get in touch with your IT people with the results if you get different outcomes.

[J111]

ok Thanks for the advice.

[J111]

Hello,

The administrator checked and the WORK space was fine during the run.

Seems like not a code problem either.

any idea ?