BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
max64
Fluorite | Level 6

Hello.

 

I'm using the ATSM package of the Proc TSMODEL. I’d like to know if the EXTREME string of the PREFILTER option of the DIAGSPEC.SetOption Method, is equivalent to the OUTLIER OPTION of the ARIMAX statement of the PROC HPF DIAGNOSE. Here below the 2 statements (the first for the PROC TSMODEL and the second for the PROC HPFDIAGNOSE).

 

 

Many thanks in advance

 

 

1 ACCEPTED SOLUTION

Accepted Solutions
rselukar
SAS Employee

Someone else might give a better response than this but in the meantime, here is what I would say:

The diagspec object has a dedicated setARIMAXOutlier method that is closer to the OUTLIER option in the ARIMAX statement of HPFDIAGNOSE procedure.  The EXTREME string of the PREFILTER option is associated with a quick way to preprocess the series for diagnosis.  It should be the same as the PREFILTER=EXTREME option in the HPFDIAGNOSE procedure (in its PROC statement).

View solution in original post

6 REPLIES 6
rselukar
SAS Employee

Someone else might give a better response than this but in the meantime, here is what I would say:

The diagspec object has a dedicated setARIMAXOutlier method that is closer to the OUTLIER option in the ARIMAX statement of HPFDIAGNOSE procedure.  The EXTREME string of the PREFILTER option is associated with a quick way to preprocess the series for diagnosis.  It should be the same as the PREFILTER=EXTREME option in the HPFDIAGNOSE procedure (in its PROC statement).

max64
Fluorite | Level 6
Thank you so much for your gentle answer!
rselukar
SAS Employee

I am glad and you are very welcome.

max64
Fluorite | Level 6
  1. I'm using the ATSM package of Proc TSMODEL (Visual Forecasting release 8.2).
  2. I have to process about 8.000 series.
  3. The problem is that Proc Tsmodel final report, says that 8.000 series were processed but actually I get in the output datasets, forecasts and estimates for only 1200-1300 series.
  4. The number and type of the processed series, vary according to the run and are never the same.
  5. The same code applied to a different data set with different series doesn't give any problem
    %let sel_in0   =lscas.level_rp;
    %let sel_by    =region prefecture;
    %let sel_suf   =rp_dl;
    %let sel_target=avg_download_kbps;
    %let fore_x     =num_of_cells2G3G added_cells2G3G num_of_cells4G added_cells4G num_of_cells_tot added_cells_tot 
                    weekendgg holidaygg pontegg festivi gg lavorativi;
    %let end_fit   ="01jul2018"d;
    %let sel_back  =6;
    %let sel_lead  =18;
    
    %let sel_in=&sel_in0.;
    
    data _null_;
     array conta (*) &fore_x.;
     call symput('nvar', compress(dim(conta)));
     vars="&fore_x.";
     do i=1 to dim(conta);
      var=scan(vars,i," ");
      call symput('var' || compress(i),compress(var)); 
     end;
    run;
    
    %macro atsm;
     proc tsmodel
      data=&sel_in.
      outobj= (
               outfor       = LSCAS.&sel_suf._LS_OUTFOR   /* actual and forecast value*/
               outstat      = LSCAS.&sel_suf._LS_OUTSTAT   /* forecast stat and perfomance */
               outmodelinfo = LSCAS.&sel_suf._LS_MODELINFO /* selected model summary */
               parmest      = LSCAS.&sel_suf._LS_PARMEST   /* selected model details */
              ) 
      seasonality=12;
     
      *define time series ID variable and the time interval;
      id date interval = MONTH FORCEINPUTFORMAT SETMISSING=missing;
    
      *define time series and the corresponding accumulation methods;
      var &sel_target. / accumulate=average;
     
      %do v=1 %to &nvar.;
       var &&var&v.. / accumulate=average;
      %end;
    
      *define the by variables if exist;
      by &sel_by.;
    
      *using the ATSM (Automatic Time Series Model) package;
      require atsm;
    
      *starting user script;
      submit;
       *DECLARING THE ATSM OBJECTS;
       declare object dataframe(tsdf)   ; /* TSDF: Time series data frame used to group series variables for DIAGNOSE and FORENG objects */
       declare object diagspec(diagspec); /* DIAGSPEC: Diagnostic control options for DIAGNOSE object */
       declare object diagnose(diagnose); /* DIAGNOSE: Automatic time series model generation */
       declare object forecast(foreng)  ; /* FORENG:   Automatic time series model selection and forecasting */
      
       declare object outfor(outfor);             /* OUTFOR  : Collector for FORENG forecasts */
       declare object outstat(outstat);           /* OUTSTAT : Collector for FORENG forecast performance statistics */
       declare object outmodelinfo(outmodelinfo); /* OUTMODEL: Collector for selected model summary */
       declare object parmest(outest);            /* PARMEST : Collector for selected model details */
    
       array perror[2] / nosymbols; /* MINIC (ARMA selection) parameters array */ 
       array p[2]      / nosymbols; /* AR components parameters array */
       array q[2]      / nosymbols; /* MA components parameters array */
       array ps[2]     / nosymbols; /* Seasonal AR components parameters array */
       array qs[2]     / nosymbols; /* Seasonal MA components parameters array */
       array xnum[2]   / nosymbols; /* Transfer Function AR components parameters array */
       array xden[2]   / nosymbols; /* Transfer Function MA components parameters array*/
       array diff[2]   / nosymbols; /* integration test*/
       array shift[2]  / nosymbols; /* level shift min duration*/
            
       *INITIALIZE THE TSDF OBJECT AND ASSIGN THE TIME SERIES ROLES;
       rc = dataframe.initialize();
       /* Target series */
       rc = dataframe.addY(&sel_target.); 
       /* Explantory series */
       %do v=1 %to &nvar.;
        rc = dataframe.addX(&&var&v.., 'extend','LAST','required','MAYBE','keep',0,'nodiff',0,'control',1,'repmiss',1);
       %end;
            
       *OPEN THE DIAGSPEC OBJECT AND ENABLE ESM AND ARIMAX MODEL CLASS FOR DIAGNOSE;
       rc = diagspec.open();
      
       rc = diagspec.setOption(
                                'SIGLEVEL', 0.05       /* Significance level */
                               ,'CRITERION','GMAPE'    /* model selection criterion to select the best mode */
                               ,'INPUTMISSPCT',1       /* accetable size (%)of the missing observation of the input time se-ries*/
                               ,'ENTRYPCT', 0.1        /* min percentage increment of the criterion between two candidate models to stop */
                               ,'PREFILTER','EXTREME'  /* smoothed values for missing and extreme values prior to diagnostic tests*/
                               ,'TESTINPUT','BOTH'     /* log transform and trend testing of the input variables */
                               ,'SELECTINPUT','SELECT' /* maximum number of the input variables to select */
                              );
                                               
       PERROR[1]=0;PERROR[2]=24; /* AR order range for MINIC (ARMA selection) error */
       P[1]     =0;P[2]     =12; /* Nonseasonal AR order range */
       Q[1]     =0;Q[2]     =12; /* Nonseasonal MA order range */
       PS[1]    =0;PS[2]    =2;  /* Seasonal AR order range */
       QS[1]    =0;QS[2]    =2;  /* Seasonal MA order range */
       XNUM[1]  =0;XNUM[2]  =12; /* Transfer Funtcion Numerator order range */
       XDEN[1]  =0;XDEN[2]  =12; /* Transfer Funtcion Denominator order range */
       DIFF[1]  =0;DIFF[2]  =12; /* integration test */
       SHIFT[1]  =2;SHIFT[2]=6 ; /* integration test */
      
       rc=diagspec.setTrend('SIGLEVEL', 0.05, 'DIFFRAN',DIFF, 'SDIFF','AUTO');
    
       rc = diagspec.setARIMAX(
                                'SIGLEVEL', 0.2  /* ARIMAX parameters significance*/
                               ,'IDENTIFY','BOTH' /* Identification order (ARIMA, REG, or BOTH) */
                               ,'NOINT',0         /* Suppress constant term (0 | 1) */
                               ,'PERROR',PERROR   /* AR order range for MINIC (tentative ARMA order selection) error */
                               ,'P',P             /* Nonseasonal AR order range */
                               ,'Q',Q             /* Nonseasonal MA order range */
                               ,'PS',PS           /* Seasonal AR order range */
                               ,'QS',QS           /* Seasonal MA order range */
                               ,'XNUM',XNUM       /* Transfer Funtcion Numerator order range */
                               ,'XDEN',XDEN       /* Transfer Funtcion Denominator order range */
                               ,'CRITERION','SBC' /* Identification criterion */
                               ,'ESTMETHOD','CLS' /* ARIMA estimation method */
                               ,'METHOD','MINIC'  /* Tentative ARMA orders */
                               );
       /* outliers detections */
       rc=diagspec.setARIMAXOutlier(
                                     'ALLOWAO',1
                                    ,'ALLOWLS',1
                                    ,'ALLOWTLS',1
                                    ,'DETECT','MAYBE'
                                    ,'ENTRYPCT',0.1
                                    ,'FILTER','FULL'
                                    ,'MAXPCT',5
                                    ,'SIGLEVEL',0.01
                                    ,'TLSVALS',SHIFT);
       /* refine parameters of selected model excluding outliers */
       rc = diagspec.setARIMAXRefine(
                                      'SIGLEVEL',0.4
                                     ,'ORDER','ARMA:INPUT');
       /* UCM */
       rc = diagspec.setUCM( 'SIGLEVEL', 0.05
                            ,'IRREGULAR',1
                            ,'LEVEL',1
                            ,'SLOPE',1
                            ,'SEASON',1
                            ,'CYCLE',1
                            ,'AUTOREG',1
                            ,'DEPLAG',1
                           );
       /* refine parameters of selected model excluding outliers */
       rc=diagspec.setUCMRefine('SIGLEVEL',0.4,'ORDER','INPUT');
    
       /* cominbed model specification */
       rc=diagspec.setCombine();  
    
       rc = diagspec.Close();
    
       *SET THE DIAGNOSE OBJECT USING THE DIAGSPEC OBJECT AND RUN THE DIAGNOSE PROCESS;
       rc = diagnose.initialize(dataframe);
       rc = diagnose.setSpec(diagspec);                /* use diagspec defined above to run diagnose */
       rc = diagnose.SetOption('HOLDOUT', &sel_back.); /* number of periods to be used only for model selection, not identification */
       rc = diagnose.SetOption('BACK'   , &sel_back.); /* number of periods to be used for forecast (as if actuals does not exist) */
       rc = diagnose.run();
    
       *INITIALIZE THE FORENG OBJECT WITH THE DIAGNOSE RESULT AND RUN MODEL SELECTING AND GENERATE FORECASTS;
       rc = forecast.initialize(diagnose); /* use diagnose result for forecasting */
       rc = forecast.setOption('ALPHA', 0.05);    /* fit statistics*/
       rc = forecast.setOption('CRITERION','GMAPE');    /* fit statistics*/
       rc = forecast.SetOption('HOLDOUT', &sel_back.); /* number of periods to be used only for model selection, not identification */
       rc = forecast.SetOption('BACK', &sel_back.);    /*  number of periods to be used for forecast (as if actuals does not exist) */
       rc = forecast.setOption('LEAD', &sel_lead.);    /* number of periods (from back) for which has to be calculate forecast */
       rc = forecast.setOption('MINOBS.TREND',12);     /*no trend model is fitted to any series with fewer nonmissing obs */
       rc = forecast.setOption('MINOBS.SEASON',24);    /*no season model is fitted to any series with fewer nonmissing obs */
       rc=  forecast.setOption('TASK','FORECAST');     /* SELECT=model selection, estimates parameters,produces forecasts
                                                          FIT=estimates parameters by using the model specified then forecasts
                                                          UPDATE=estimates parameters by using the model specified then forecasts.
                                                                 UPDATE differs from FIT, parameters found in specified are used 
                                                                 as starting values in estimation
                                                          FORECAST=forecasts using model and parameters specified*/
       rc = forecast.run();
    
       *COLLECT THE FORECAST AND STATISTIC-OF-FIT FROM THE FORGEN OBJECT RUN RESULTS;
       rc = outfor.collect(forecast);       /* OUTFOR  : load forecasts */
       rc = outstat.collect(forecast);      /* OUTSTAT : load forecast performance statistics */
       rc = outmodelinfo.collect(forecast); /* OUTMODEL: load selected model summary */
       rc = parmest.collect(forecast);      /* PARMEST : load selected model details */
      endsubmit;
     quit;
    %mend atsm;
    
    option mprint;
    %atsm;
     
dw_sas
SAS Employee

This new issue is something that will likely need to be investigated by Tech Support.  Please open a Tech Support track using the information in the link below:

 

https://support.sas.com/en/technical-support/contact-sas.html

 

In addition to the code included in your recent posting, please also provide the following information to Tech Support:

 

1)  Sample data which reproduces the behavior you are observing,

2)  Details about your CAS session setup, such as the number of nodes, cores per node, etc.  (Note:  This information can be found in the listing file generated by PROC TSMODEL.)

 

Thank you!

DW

max64
Fluorite | Level 6
Thank you so much for your attention!

SAS Innovate 2025: Save the Date

 SAS Innovate 2025 is scheduled for May 6-9 in Orlando, FL. Sign up to be first to learn about the agenda and registration!

Save the date!

Multiple Linear Regression in SAS

Learn how to run multiple linear regression models with and without interactions, presented by SAS user Alex Chaplin.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 6 replies
  • 2598 views
  • 1 like
  • 3 in conversation