topic Re: SAS Not responding in Statistical Procedures

SAS Not responding

Barkamih — Sat, 14 Apr 2018 22:04:52 GMT

Hi Guys

I have an issue when I'm running this code of my dataset. which sas giving (Not responding) after a few minutes by running this code, while when I was running the same code for small dataset had great results. Dataset has 218006 observations and 28 variables.

What do you think the best way to solve this problem?

PROC SORT DATA = NEW;
BY  cow_id pr;
RUN;

proc nlin data = new outest=ESTIMATES;
parms A = 15 B = 0.19 C = -0.0012 ;
bounds  A B C > 0; 
by cow_id pr;
model  TEST_DAY_MILK_KG = A * Time **b * exp(-C*Time);
output out = Fit predicted = Pred ;
run;


proc sql;
create table persistency as
select 
cow_id,pr,
A, B, C, 
-(B+1)*LOG(C) as P,
A * (B/C)**(B) * exp(-B) as peakYield,
(B/C) as tmdays
from ESTIMATES
where _TYPE_ = "FINAL";
select * from persistency;
run;
quit;

data merged;
merge new persistency;
by cow_id pr;
run;

PROC SORT DATA = MERGED;
BY cow_id time ;
RUN;

Re: SAS Not responding

thesasuser — Sun, 15 Apr 2018 03:02:45 GMT

Please run each of the procs / data step one by one.
You can identify the error.
Then post the logs

Re: SAS Not responding

Kurt_Bremser — Sun, 15 Apr 2018 06:45:02 GMT

This part:

select * from persistency;
run;

sends all data from the dataset to the output window, which can take very long in a client/server environment and cause the client (EG) to hang/crash in some cases because of running out of memory.

Re: SAS Not responding

Barkamih — Sun, 15 Apr 2018 08:55:21 GMT

this part takes a long time and then following by not responding prosses, and I can't reach the log file of the software or any tool of the software, which forces me to end the program.

proc nlin data = new outest=ESTIMATES;
parms A = 15 B = 0.19 C = -0.0012 ;
bounds  A B C > 0; 
by cow_id pr;
model  TEST_DAY_MILK_KG = A * Time **b * exp(-C*Time);
output out = Fit predicted = Pred ;
run;

Re: SAS Not responding

thesasuser — Sun, 15 Apr 2018 10:31:13 GMT

Hello
You have defined c =-0.0012
Again in the bounds A B C>0
These two statements are contradictory.
Remove the negative sign. Take C=0.0012
There is already a negative sign in the exp part.
Please try this and let us know
Please don’t forget to put
endsas; before proc sql.
Pl also take care of the suggestion by Kurt.

You can simplify your model by linearizing it.

Re: SAS Not responding

Barkamih — Sun, 15 Apr 2018 20:51:24 GMT

Unfortunately, nothing change. I'm still having the same issue.

I will try much more to see what will happen.

thank you so much for your time that you spent to help me out with that problem, and I will keep in touch with you if I found any solution

Regards

Re: SAS Not responding

thesasuser — Mon, 16 Apr 2018 01:31:25 GMT

Further to what I have said,. you should reconsider your model.
TEST_DAY_MILK_KG = A * Time **b * exp(-C*Time);

At Time=0 (start)

Time**b=0

exp(-C*Time) =exp(0) =1

Thus TEST_DAY_MILK_KG = A *(0)*(1) =0

At Time = Infinity (After a large time)

Time **b will be very large or infinity

exp(-C * Time)=1/exp(C*Time)=1/infinity =0

Thus TEST_DAY_MILK_KG = A *(Infinity)*0 = 0

Though I understand modeling, I do not know about your process.

Re: SAS Not responding

Rick_SAS — Mon, 16 Apr 2018 18:25:09 GMT

> What do you think the best way to solve this problem?

I think the best way to solve this equation is to transform it to a LINEAR system. If

M = A * t**B * exp(-C*t)

then

log(M) = log(A) + B*log(t) - C*t;

Therefore, in the DATA step define

logM = log(TEST_DAY_MILK_KG);

logT = log(Time);

Then use PROC REG to solve the linear system. If desired, you can transform the parameter estimates and predictions back to their original scale, as follows:

/* simulate data */
data new;
do cow_id=1,2; pr=1;
   do Time = 1 to 500 by 7;
      logM = log(15) + 0.2*log(Time) - 0.0012*Time + rand("Normal",0,0.05);
      TEST_DAY_MILK_KG = exp(logM);
      output;
   end;
end;
keep TEST_DAY_MILK_KG Time cow_id pr;
run;
/* end simulation */
/************************************/

/* take LOG transform of the data */
data LOG;
   set new;
   logM = log(TEST_DAY_MILK_KG);
   logT = log(Time);
run;

/* linear regression */
proc reg data=LOG noprint outest=LogESTIMATES;
   by cow_id pr;
   model  logM = logT Time;
   output out = LogFit predicted = LogPred;
run;

/* transform estimates to original scale */
data Estimates;
   set LogESTIMATES;
   A = exp(Intercept);
   B = logT;
   C = -Time;
run;

/* transform predictions to original scale */
data Fit;
   set LogFit;
   Pred = exp(LogPred);
run;

/* graph data; overlay fit */
proc sgplot data=Fit;
   where cow_id=1 and pr=1;
   scatter x=Time y=TEST_DAY_MILK_KG;
   series x=Time y=Pred;
run;

Re: SAS Not responding

Barkamih — Mon, 16 Apr 2018 18:10:23 GMT

thanks for your help.

But I have more questions about this part of the code. I want to keep all cow_id data, this one just presented two cow_ids and I have a lot in my dataset.

do cow_id=1,2; pr=1;

also, I want to keep all variables that presented in data new.

keep TEST_DAY_MILK_KG Time cow_id pr;

So, how it could look like this code?

thanks again

data new;
do cow_id=1,2; pr=1;
   do Time = 1 to 500 by 7;
      logM = log(15) + 0.2*log(Time) - 0.0012*Time + rand("Normal",0,0.05);
      TEST_DAY_MILK_KG = exp(logM);
      output;
   end;
end;
keep TEST_DAY_MILK_KG Time cow_id pr;
run;

Re: SAS Not responding

Rick_SAS — Mon, 16 Apr 2018 18:24:37 GMT

Ignore the 'DATA new' step. That was just to simulate some data so that I could demonstrate the technique. Use the code that begins with the long 'divider' comment /*************************************/