topic Accessing standardized residuals: arima in SAS Forecasting and Econometrics

Accessing standardized residuals: arima

sasalex2024 — Thu, 19 Dec 2024 22:08:47 GMT

Dear SAS Community,

Suppose I have standard PROC ARIMA commands like this:

proc arima data=a;

identify var=Yt crosscorr=(X1t X2t) noprint;

estimate q=1 input=( (1) X1t (1) X2t ) method=ml;

run;

The command displays a QQ plot of standardized residuals. However, if I want to access the values of those residuals, could you please guide me on how to do it?

I have tried using the outmodel etc, but I couldn't find a way to extract the residuals.

Thank you!

Re: Accessing standardized residuals: arima

sbxkoenk — Fri, 20 Dec 2024 17:05:28 GMT

The below code provides you with the (X,Y)-coordinates of the markers in the QQ-plot (see data set named "work.abc"), but the data are sorted. I don't know how you can find out which observation belongs to which time stamp in your time series.

title1 'Simulated IMA(1,1) Series';
data a;
  u1 = 0.9; a1 = 0;
  do i = -50 to 100;
     a = rannor( 32565 );
     u = u1 + a - .8 * a1;
     if i > 0 then output;
     a1 = a;
     u1 = u;
  end;
run;

*ods trace on; ods trace off;
ODS output ResidualNormalityPanel=work.abc;
/*-- Simulated IMA Model --*/
proc arima data=a;
  identify var=u;
  run;
  identify var=u(1);
  run;
  estimate q=1 ;
  run;
quit;
/* end of program */

Ciao, Koen

Re: Accessing standardized residuals: arima

sasalex2024 — Fri, 20 Dec 2024 21:12:42 GMT

Dear Koen,

Thank you very much for your response. Your code works perfectly for the example model you provided. However, I have been unable to adapt it to my specific case, which involves extracting residuals from a model with inputs and ARMA errors. I tried including the lines:

*ods trace on; ods trace off;
ODS output ResidualNormalityPanel=work.abc;

before my PROC ARIMA code and ending with a quit; after the last run;. Unfortunately, this did not produce the dataset work.abc as expected. Would you be able to clarify if additional modifications are needed to make this approach work for models with inputs and ARMA errors? Thank you once again for your time and assistance!

Re: Accessing standardized residuals: arima

ballardw — Fri, 20 Dec 2024 21:50:29 GMT

@sasalex2024 wrote:

Dear Koen,

Thank you very much for your response. Your code works perfectly for the example model you provided. However, I have been unable to adapt it to my specific case, which involves extracting residuals from a model with inputs and ARMA errors. I tried including the lines:
*ods trace on; ods trace off; ODS output ResidualNormalityPanel=work.abc;
before my PROC ARIMA code and ending with a quit; after the last run;. Unfortunately, this did not produce the dataset work.abc as expected. Would you be able to clarify if additional modifications are needed to make this approach work for models with inputs and ARMA errors? Thank you once again for your time and assistance!

Run the trace on/off around your model code and show us the LOG. The log will include a bunch of bits with Names of the ODS objects created, include all of that along with the code and other messages the procedure generates. It may be that your install created a different ODS object than the ResidualNormalityPanel, or the spelling changed, from @sbxkoenk's install.

ODS TRACE ON;
proc arima data=a;
identify var=Yt crosscorr=(X1t X2t) noprint;
estimate q=1 input=( (1) X1t (1) X2t ) method=ml;
run;
quit;

ODS TRACE OFF;

Re: Accessing standardized residuals: arima

sasalex2024 — Fri, 20 Dec 2024 21:50:45 GMT

Hi, thank you. This is the log:

1 OPTIONS NONOTES NOSTIMER NOSOURCE NOSYNTAXCHECK;

NOTE: ODS statements in the SAS Studio environment may disable some output features.

70 ODS TRACE ON;

71 proc arima data=series_31;

72 identify var=Yt(1) crosscorr=(X1t(1) X2t(1) X3t(1)) noprint;

73 estimate q=1 input=( (1) X1t (1) X2t (1) X3t) method=ml;

74 run;

NOTE: The ODS graphics associated with the IDENTIFY statement is suppressed because of the presence of NOPRINT option.

Output Added:

-------------

Name: ParameterEstimates

Label: Maximum Likelihood Estimation

Template: ets.Arima.ParameterEstimates

Path: Arima.Estimate.ParameterEstimates

-------------

Output Added:

-------------

Name: FitStatistics

Label: Fit Statistics

Template: ets.Arima.FitStatistics

Path: Arima.Estimate.FitStatistics

-------------

Output Added:

-------------

Name: CorrB

Label: Correlations of Parameter Estimates

Template: ets.Arima.CorrB

Path: Arima.Estimate.CorrB

-------------

Output Added:

-------------

Name: ChiSqAuto

Label: Autocorrelation Check of Residuals

Template: ets.Arima.ChiSqAuto

Path: Arima.Estimate.ChiSqAuto

-------------

Output Added:

-------------

Name: ResidualCorrPanel

Label: Residual Correlation Panel

Template: ets.arima.Graphics.CorrPanel

Path: Arima.Estimate.ResidualCorrPanel

-------------

Output Added:

-------------

Name: ResidualNormalityPanel

Label: Residual Normality Panel

Template: ets.arima.Graphics.NormalityPanel

Path: Arima.Estimate.ResidualNormalityPanel

-------------

Output Added:

-------------

Name: ModelDescription

Label: Model for variable Yt

Template: ets.Arima.ModelDescription

Path: Arima.Estimate.Filter.ModelDescription

-------------

Output Added:

-------------

Name: MAPolynomial

Label: Moving Average Factors

Template: ets.Arima.Equation

Path: Arima.Estimate.Filter.MAPolynomial

-------------

Output Added:

-------------

Name: InputDescription

Label: Input Number 1

Template: ets.Arima.InputDescription

Path: Arima.Estimate.Filter.InputDescription

-------------

Output Added:

-------------

Name: NumPolynomial

Label: Numerator Factors

Template: ets.Arima.Equation

Path: Arima.Estimate.Filter.NumPolynomial

-------------

Output Added:

-------------

Name: InputDescription

Label: Input Number 2

Template: ets.Arima.InputDescription

Path: Arima.Estimate.Filter.InputDescription

-------------

Output Added:

-------------

Name: NumPolynomial

Label: Numerator Factors

Template: ets.Arima.Equation

Path: Arima.Estimate.Filter.NumPolynomial

-------------

Output Added:

-------------

Name: InputDescription

Label: Input Number 3

Template: ets.Arima.InputDescription

Path: Arima.Estimate.Filter.InputDescription

-------------

Output Added:

-------------

Name: NumPolynomial

Label: Numerator Factors

Template: ets.Arima.Equation

Path: Arima.Estimate.Filter.NumPolynomial

-------------

75 quit;

NOTE: PROCEDURE ARIMA used (Total process time):

real time 0.25 seconds

user cpu time 0.13 seconds

system cpu time 0.02 seconds

memory 15596.43k

OS Memory 36812.00k

Timestamp 12/20/2024 09:47:05 PM

Step Count 57 Switch Count 12

Page Faults 0

Page Reclaims 8661

Page Swaps 0

Voluntary Context Switches 531

Involuntary Context Switches 8

Block Input Operations 0

Block Output Operations 952

77 ODS TRACE OFF;

79 OPTIONS NONOTES NOSTIMER NOSOURCE NOSYNTAXCHECK;

------------------

This is the code:

ODS TRACE ON;
proc arima data=series_31;
identify var=Yt(1) crosscorr=(X1t(1) X2t(1) X3t(1)) noprint;
estimate q=1 input=( (1) X1t (1) X2t (1) X3t) method=ml;
run;
quit;

ODS TRACE OFF;

Re: Accessing standardized residuals: arima

ballardw — Fri, 20 Dec 2024 22:08:42 GMT

This part of your log:

Output Added:
-------------
Name: ResidualNormalityPanel
Label: Residual Normality Panel
Template: ets.arima.Graphics.NormalityPanel
Path: Arima.Estimate.ResidualNormalityPanel
-------------

Looks like the ods object you want is the same as @sbxkoenk's code used.

Perhaps you can show the Log where you have the ODS OUTPUT statement with your model.

proc arima data=series_31;
   identify var=Yt(1) crosscorr=(X1t(1) X2t(1) X3t(1)) noprint;
   estimate q=1 input=( (1) X1t (1) X2t (1) X3t) method=ml;
   ODS OUTPUT ResidualNormalityPanel= work.yourdatasetname ;
run;
quit;

I don't have access to Proc Arima so can't run the code. There might be that with procedures supporting run group processing like Proc Arima that the location of the ODS OUTPUT has to be within the run group.

Re: Accessing standardized residuals: arima

sbxkoenk — Fri, 20 Dec 2024 22:11:12 GMT

This is a Regression with ARMA Errors

I combine input series with an ARMA model for the error.

The following statements regress SALE on DISCOUNT and PRICE but with the error term of the regression model (called the noise series in ARIMA modeling terminology) assumed to be an ARMA(1,1) process:

ods output ResidualNormalityPanel=work.abc;
proc arima data=sashelp.pricedata;
   where region=1 AND line=1 and product=1;
   identify var=sale crosscorr=(price discount);
   estimate p=1 q=1 input=(price discount);
run;
QUIT;

It works !

Koen

Re: Accessing standardized residuals: arima

sasalex2024 — Sat, 21 Dec 2024 15:32:38 GMT

Thank you very much for your help. I ran the following code based on yours (shown below this message), and although the log shows an error: "ERROR: Variable region is not on file WORK.SERIES_31," it still produces the abc table in the Work directory.

I apologize for troubling you again, but there is a column in that table (the 10th column, I believe) named "residual." I assume these are the residuals from the model, correct? And they are sorted I suppose. If so, could you please tell me how I can obtain these residuals in their original, "unsorted" order? I also need to plot them over time.

Thank you again!

----

proc arima data=series_31;
where region=1 AND line=1 and product=1;
identify var=Yt(1) crosscorr=(X1t(1) X2t(1) X3t(1)) noprint;
estimate q=1 input=( (1) X1t (1) X2t (1) X3t) method=ml;

run;
quit;

Re: Accessing standardized residuals: arima

sasalex2024 — Sat, 21 Dec 2024 15:38:17 GMT

Thank you so much, I have run your code and it works too! It produces the same values as in the method given by Koen. Again, thank you very much.

Re: Accessing standardized residuals: arima

sbxkoenk — Sat, 21 Dec 2024 16:19:42 GMT

@sasalex2024 wrote:

... If so, could you please tell me how I can obtain these residuals in their original, "unsorted" order?
I also need to plot them over time.

Here is how to get the residuals in their original "unsorted" order.
By the way ... if you are just interested in the residuals, then use the FORECAST statement with OUT= option. There's no need to get the residuals from the ResidualNormalityPanel (work.abc) dataset.

data work.have;
 set sashelp.pricedata;
 idobs=_N_;
run;

ods output ResidualNormalityPanel=work.abc;
proc arima data=work.have;
   where region=1 AND line=1 and product=1;
   identify var=sale crosscorr=(price discount);
   estimate p=1 q=1 input=(price discount);
   forecast lead=0 id=idobs interval=month out=work.xyz NOPRINT;
run;
QUIT;

PROC SQL noprint;
 create table work.want as
 select t1.* , t2.*
 from   work.xyz t1
      , work.abc t2
 where put(t1.residual,10.6) = put(t2.residual,10.6)
 order by idobs;
QUIT;
/* end of program */

To plot a time series, see here :

Plotting Time Series Data - SAS Support Communities

Ciao, Koen

Re: Accessing standardized residuals: arima

sasalex2024 — Sat, 21 Dec 2024 16:51:39 GMT

Thank you very much for your lightning-fast response! Amazing how it works elegantly! Thanks again.

Re: Accessing standardized residuals: arima

sasalex2024 — Sun, 22 Dec 2024 01:05:21 GMT

Dear Koen,

I also tried the code below, do yo think it is correct and consistent with yours? (even though I am sure it is not as efficient as yours). I've created standardized residuals by dividing the residuals by Std Error Estimate. (I have Time column in my original data.) The code is below, thank you very much again.

proc arima data=series_31;
identify var=Yt(1) crosscorr=(X1t(1) X2t(1) X3t(1)) noprint;
estimate q=1 input=( (1) X1t (1) X2t (1) X3t) method=ml outcorr outest=est3 outmodel=model3;
forecast lead=0 interval=month id=Time out=results;
run;

/* Creating standardized residuals */
data results;
set results;
st_resid = residual / 23.79973;
run;

/* Plotting st. residuals over time */
proc sgplot data=results;
vbar Time / response=st_resid;
xaxis label="Time";
yaxis label="Standardized Residuals";
run;

Re: Accessing standardized residuals: arima

sbxkoenk — Sun, 22 Dec 2024 14:40:26 GMT

/* You should avoid hard-coding (23.79973). */
/* Just divide by (constant) STD variable.  */

proc arima data=work.have;
   where region=1 AND line=1 and product=1;
   identify var=sale crosscorr=(price discount);
   estimate p=1 q=1 input=(price discount);
   forecast lead=0 id=date interval=month out=work.results NOPRINT;
run;
QUIT;

/* Creating standardized residuals */
data results;
 set results;
 st_resid = (residual / std);
run;

ods graphics on / width=1000px height=700px;
/* Plotting st. residuals over time */
proc sgplot data=results;
 vbar date / response=st_resid;
 xaxis label="Time";
 yaxis label="Standardized Residuals";
run;
/* end of program */

Koen