Unsmooth time series data

[q5pham]

Hi everyone,

Could anyone guide me on steps how to unsmooth a time serie data and which procedures can do the task on SAS? As far as I undestand, I would first need to identify the autocorrelation level and unsmooth in the next step.

*1st step: I suppose Proc ARIMA is the one that can estimate smoothing coefficients of my data through the esitmation the orders of AR and MA. I tried to run my tested code:

ods exclude all;

proc arima data=Sample ;
    identify var=performance centre MINIC SCAN  ;
    by code;
    ods output DescStats=DescStat TentativeOrders=orders;
run ;

but there is an ERROR: The variance of the time series for variable Performance is smaller than 1E-12. I successfully run in my first trial, but for some reason there were errors in my next trials. My data and variable is unchanged.

*2nd step: To remove the autocorrelation or unsmooth the data - Is there any built-in proc in SAS to do the task?

Your help is greatly appreciated as I've been struggling with this for a week now.

Thanks, Mai

[q5pham]

Found the problem. The QUIT statement is missing and it creates a big burden to me.

My next question is my data is relatively big, contains more than 5000 groups (the variable code). And SAS can only handle upto some few dozen tables. Is there any way to estimate the theta coefficients of the MA(2) process with my data.

Thanks

[SteveDenham]

Efficient use of ODS OUTPUT and ODS EXCLUDE statements may make what you are asking for much easier.

Steve Denham

[q5pham]

Hi Steve,

I do use ODS Exclude and Ods Output. My next problem arise. Proc Arima is just a mystery that I cannot figure out what's really happening. I encountered the error: "The variance of the time series for variable Performance is smaller than 1E-12" again. If I run without the BY statement, it perfectly works. Below is my log. Hope you could help:

      ods exclude all;
29         ods select InitialMAEstimates;
30         proc arima data=finalsample8 ;
31             by code;
32             identify var=performance center nlag=12 ;
33             estimate q=(1)(1) noconstant ;
34             ods output PrelimEstimates (match_all=q persist=proc)  =MA ;
NOTE: MATCH_ALL=Q is no longer needed if your goal is to create a single data set containing all the variables in all the
      identified objects. By default, if the ODS OUTPUT statement identifies multiple output objects, ODS now combines all the
      objects into a single data set that contains all the variables from all the objects.
35         run ;

NOTE: Interactivity disabled with BY processing.
36         quit;

ERROR: The variance of the time series for variable Performance is smaller than 1E-12.
ERROR: Estimation was not performed because identification was not done.
NOTE: The above message was for the following BY group:
      Code=12406
WARNING: Output 'InitialMAEstimates' was not created.  Make sure that the output object name, label, or path is spelled correctly.  
         Also, verify that the appropriate procedure options are used to produce the requested output object.  For example, verify
         that the NOPRINT option is not used.
NOTE: PROCEDURE ARIMA used (Total process time):
      real time           3:51.11
      cpu time            1:17.81

[Rick_SAS]

Sometimes errors like this occur when the BY variable has a level for which there are only one or two values, or for which the data are constant. Use PROC FREQ to examine your BY-group variable:

proc freq data=finalsample8 ;

tables code;

run;

This will identify the first BY group and tellyou how many observations. If the first BY group has more than a few observations, look at the 'performance variable':

proc means data=finalsample8 N NMISS MEAN STD;

WHERE code = /* put value of first level here */;

var performance;

run;

[q5pham]

Hi Rick,

My observations in each group are large, min 30 and max 140+.

I cannot understand why, but after I change the output table from InitialMAEstimates to MAPolynomial, the error diappears. I suspected it's the error on the table name, instead of the small variance.

But it's another mystery. When I ran ods trace on, the InitialMAEstimates did not show up, which means there is no table with that name. But it is listed on https://support.sas.com/documentation/cdl/en/etsug/60372/HTML/default/viewer.htm#etsug_arima_sect052...

Now, instead of producing 5000 MAPolynomial tables corresponding to 5000 groups, it only produced 500, the maximum level I can set in the Tools/options/Results/General Results menu.

Basically, I would like to estimate the theta coefficients of the MA(2) process. If there are any other suggestion other than Proc Arima, I would really appreciate.

Thanks, Mai

[Rick_SAS]

Comment out the BY statement. Replace it with

WHERE code = /* first level here */;

Debug the code in this simple case.

In this simpler situation, I suggest that comment out the ODS SELECT statement and turn on ODS TRACE ON;

[q5pham]

Yeap. I'm testing with one code only for now, using the where code=. It works. But i still have to work out with the remaining 5000 codes