topic Re: Data size limitations for running code in SAS Programming

Data size limitations for running code

J111 — Mon, 11 Aug 2025 05:58:48 GMT

Hello,

Are there data size limitations for the following code,

if so is there a solution rather than a smaller data input:

----------------------------------------------------

filename sas_code temp ;

data _null_ ;

Set MATCHING_TABLE_FUNC end = last ;

file sas_code ;

if _N_ = 1 then put "options ls = 256 errors = max nospool ; Data Formula_ ; set TRNSP_SERIES_LIST ;" ;

put Target '=' Formula ";" ;

if last then put "Run ;" ;

run ;

data _null_ ;

infile sas_code ;

input ;

put _infile_ ;

run ;

%include sas_code ;

----------------------

The log has an error of connection to the server was lost.

The matching table includes thousands of formula such as

target= x1 + x2

The TRNSP_SERIES_LIST SAS file includes 487 rows and 4624 columns.

If the TRNSP_SERIES_LIST includes only 175 rows there is no log error of connection and the program runs smoothly.

We work with Linux, SAS EG 8.2

-------------------------

I can invent data if it is required.

Thanks in advance.

Re: Data size limitations for running code

Ksharp — Mon, 11 Aug 2025 06:30:45 GMT

Can you post dataset "TRNSP_SERIES_LIST " ,especially variable "Target" and "Formula" .

Re: Data size limitations for running code

Kurt_Bremser — Mon, 11 Aug 2025 06:33:14 GMT

How many observations does matching_table_func have?

How are variables target and formula defined?

I would run this code with dataset option OBS=1 in the first step, to check the basic validity. From there, expand the OBS number.

Re: Data size limitations for running code

J111 — Mon, 11 Aug 2025 06:50:17 GMT

Hello,

I will try to answer your questions:-

The code is valid it runs very good for 175 rows in the TRNS_SERIES_LIST table.

The matching table includes 2454 rows each row with one target formula.

Each formula includes up to 20 variables.

The TRNS_SERIES_LIST includes one column of date and thousands columns of variables.

These variables appear in the formula.

Thanks

Re: Data size limitations for running code

J111 — Mon, 11 Aug 2025 06:59:37 GMT

Hello,

The TRNP_SERIES_LIST columns looks like this

DATE x1 x2 x3 until x4623

The x(i) are variables from TRNS_SERIES_LIST that appear in the formula in the MATCHING_TABLE.

The matching table includes target name of a new variable to calculate and the formula of variables.

for example new_var = x1 + x2 /x3 + x4

Re: Data size limitations for running code

J111 — Mon, 11 Aug 2025 07:01:34 GMT

Target and formula are character variables in the matching table.

Re: Data size limitations for running code

J111 — Mon, 11 Aug 2025 07:20:50 GMT

Hello,

The following log error

Error: The connection to the server has been lost.

appears for (obs=177)

for obs with lower values such as 170, 175, 176 the program runs fine.

------------------------------------------------------------------------------------

filename sas_code temp ;

data _null_ ;

Set MATCHING_TABLE_FUNC end = last ;

file sas_code ;

if _N_ = 1 then put "options ls = 256 errors = max nospool ; Data Formula_ ; set TRNSP_SERIES_LIST(obs=177) ;" ;

put Target '=' Formula ";" ;

if last then put "Run ;" ;

run ;

data _null_ ;

infile sas_code ;

input ;

put _infile_ ;

run ;

%include sas_code ;

Re: Data size limitations for running code

ballardw — Mon, 11 Aug 2025 07:34:37 GMT

@J111 wrote:

Hello,

The following log error

Error: The connection to the server has been lost.

appears for (obs=177)

for obs with lower values such as 170, 175, 176 the program runs fine.

Does it ALWAYS fail at OBS=177?

What is the size of the text file written?

If it repeatedly fails at the same OBS number then you may be running into a local file space limit, at in user files at that location are limited to XXXX bytes. That would indicate time to ask your admin.

If always fails at 177 have you looked at the content of OBS=177?, perhaps the values has something in it causing a problem. Make a version of the data set that drops the current observation 177 and see if that runs.

If it occurs at different observations you likely have some other network interference and should talk to your IT support.

You may also need to describe your SAS environment.

Re: Data size limitations for running code

Ksharp — Mon, 11 Aug 2025 07:44:30 GMT

You could try to use CALL EXECUTE instead of "%include" .

data MATCHING_TABLE_FUNC ;
input  Target $ Formula & $20.;
cards;
v1  x1 + x2 /x3 + x4
v2  x1 + x2 + x4
v3  x1 + x2 + x3
v4  x1 + x2 
;

data TRNP_SERIES_LIST ;
input date :date9. x1 x2 x3 x4;
format date date9.;
cards;
01jan2010 1 2 3 4
01jan2010 4 3 2 1
;

data MATCHING_TABLE_FUNC2;
 set MATCHING_TABLE_FUNC;
 function=catx('=',target,formula)||';';
 keep function;
run;
data _null_;
set MATCHING_TABLE_FUNC2 end=last;
if _n_=1 then call execute('Data Formula_ ; set TRNP_SERIES_LIST ;');
call execute(strip(function));
if last then call execute('run;');
run;

Re: Data size limitations for running code

Kurt_Bremser — Mon, 11 Aug 2025 07:45:15 GMT

Restrict the obs of the formula table:

data _null_ ; 
     Set MATCHING_TABLE_FUNC (obs=1) end = last ;
     file sas_code ; 
if _N_ = 1 then put "Data Formula_ ; set TRNSP_SERIES_LIST;" ; 
     put Target '='  Formula ";" ;
if last then put "Run ;" ; 
run ;

and see if it still fails at obs 176 of trnsp_series_list.

Re: Data size limitations for running code

Ksharp — Mon, 11 Aug 2025 07:55:39 GMT

I think ballardw was right.
There could be some error data at OBS=177.
For example:
obs x1 x2
177 1 0
when you have formula "target=x1/x2;" ,that could lead to an error.

Re: Data size limitations for running code

J111 — Mon, 11 Aug 2025 08:22:46 GMT

Hello,

I am running several combinations of obs=

for the combination below, the program runs fine.

once increase obs from 300 to 350 the connection to the server is lost.

I shall check all suggestions posted and update regarding the results.

------------------------

filename sas_code temp ;

data _null_ ;

Set MATCHING_TABLE_FUNC(obs=2500) end = last ;

file sas_code ;

if _N_ = 1 then put "options ls = 256 errors = max nospool ; Data Formula_ ; set TRNSP_SERIES_LIST(obs=300) ;" ;

put Target '=' Formula ";" ;

if last then put "Run ;" ;

run ;

data _null_ ;

infile sas_code ;

input ;

put _infile_ ;

run ;

%include sas_code ;

Re: Data size limitations for running code

Kurt_Bremser — Mon, 11 Aug 2025 08:33:35 GMT

Run this:

data _null_;
set MATCHING_TABLE_FUNC end=last;
file sas_code;
if _N_ = 1 then put "options ls = 256 errors = max nospool; Data Formula_; set TRNSP_SERIES_LIST(obs=10);"  
put Target '='  Formula ";";
if last then put "run;";
run;

%include sas_code;

and then inspect the log for possible problems with the created code.

Since you add several thousand new variables (each with 8 bytes if numeric), if the source dataset already has a large observation size you might run against a limit in your WORK.

Re: Data size limitations for running code

J111 — Mon, 11 Aug 2025 08:50:57 GMT

Hello,

I could not find code issues with obs=10

the only note is

NOTE: Mathematical operations could not be performed at the following places. The results of the operations have been set to missing values

-----

We have at least 150 GIGA space for work on the server.

----

Matching table has less than 2500 rows.

Why by limiting it to (obs=2500) the trns_table could be larger than (obs=177) ? and even (obs=300)

---

there is no consistency - next run with combination of 2500 & 300 ended with connection error.

Thanks

Re: Data size limitations for running code

Kurt_Bremser — Mon, 11 Aug 2025 09:23:52 GMT

Any diligent SAS admin will establish usage quotas on servers, to prevent one user/program from eating up everything. So your individual quota may be much smaller than the overall size of the storage volume.

But the fact that this does not happen consistently lets me suspect outside interference (network issues, router timeouts, anti-malware software etc).

Run the same identical code multiple times in succession, and get in touch with your IT people with the results if you get different outcomes.

Re: Data size limitations for running code

J111 — Mon, 11 Aug 2025 10:07:35 GMT

Hello,

The administrator checked and the WORK space was fine during the run.

Seems like not a code problem either.

any idea ?

Re: Data size limitations for running code

J111 — Mon, 11 Aug 2025 10:46:15 GMT

ok Thanks for the advice.

Re: Data size limitations for running code

Tom — Mon, 11 Aug 2025 13:25:41 GMT

@J111 wrote:

Hello,

The administrator checked and the WORK space was fine during the run.

Seems like not a code problem either.

any idea ?

What "server" are you losing connection to? Your code did not appear to be attempting to use SAS/Connect or SAS/Access to some external database.

Is it possible you are using some front end tool to submit your program, like SAS/Studio or Enterprise Guide, and it is the connection from that front end to the SAS server itself that is closing?

Perhaps it is because the LOG file is just getting too big?

Try using PROC PRINTTO to route the log for that section of the code to another file.

Perhaps it is because the job is just taking too long?

Re: Data size limitations for running code

J111 — Tue, 12 Aug 2025 09:04:02 GMT

Thanks for the proc printo idea, yes we are using EGuide

8

Re: Data size limitations for running code

Kurt_Bremser — Tue, 12 Aug 2025 10:40:57 GMT

@J111 wrote:
Thanks for the proc printo idea, yes we are using EGuide

Then you should reduce the log size; EG needs to keep the whole log in memory, and the nicely formatted text eats a lot of it.

Why do you need a complete listing of the created code in the log, when the later execution of it will be in the log anyway?