TSMODEL: using the CFC object with external forecast

[andrea_magatti]

Hi all,

I would like to use the TSM package of the TSMODEL procedure, to use the CFC Object using forecasts from external models:

The Scenario is: I have 6 NNET models, each one is forecasting the same variable with different approaches, in terms of data and architecture of the Neural Network.

I want to use some methods of the CFC object to calculate a weighted average between the six forecasts.

As far as I have understood, I could use the EXMSPEC to build a TSM object usable in CFC.

The code runs with no error, but seems like is not producing any forecast.

What am I missing?

proc tsmodel data=casuser.test
	outobj=(
		fitstats = casuser.ts_comb_fit 
		forecast = casuser.ts_comb_for
		)
		outarray = casuser.predict
		outscalar = casuser.inputs;
	id data_gas_G1  interval=day;

	/* 	inscalars &AE_list; */
	/* 	inscalars &P_List; */
	var Target;
	require tsm;
	outarrays weighted_forecast;
	outscalars p_fu p_ter;
	submit;
		declare object fitstats(tsmstat);
		declare object forecast(tsmfor);

		/* Specify external models		 */
		declare object ext_full(exmspec);
		declare object ext_terna(exmspec);
		declare object newdafne(CFC);
		declare object tsm_full(tsm);
		declare object tsm_terna(tsm);
		rc = ext_full.Open();
		rc = ext_full.SetOption(
		'Predict', 'P_Full',
		'Error', 'AE_Full');
		rc = ext_full.Close();
		rc = ext_terna.Open();
		rc = ext_terna.SetOption(
		'Predict', 'P_Terna',
		'Error', 'AE_Terna');
		rc = ext_terna.Close();
		array p_fu[1] / nosymbols;
		array ae_fu[1] / nosymbols;
		array p_ter[1] / nosymbols;
		array ae_ter[1] / nosymbols;
		p_fu[1] = P_Full;
		ae_fu[1] = AE_Full;
		p_ter[1] = P_Terna;
		ae_ter[1] = AE_Terna;
		rc = tsm_full.Initialize(ext_full);
		rc = tsm_full.AddExternal(p_fu, 'predict');
		rc = tsm_full.AddExternal(ae_fu, 'error');
		rc = tsm_terna.Initialize(ext_terna);
		rc = tsm_terna.AddExternal(p_ter, 'predict');
		rc = tsm_terna.AddExternal(ae_ter, 'error');
		rc = newdafne.Initialize();
		rc = newdafne.SetY(Target);
		rc = newdafne.AddModel(tsm_full);
		rc = newdafne.AddModel(tsm_terna);
		rc = newdafne.AddPredict(p_fu, ae_fu, 'Full');
		rc = newdafne.AddPredict(p_ter, ae_ter, 'Terna');
		rc = newdafne.Criterion('Fit');
		rc = newdafne.SetOption('weight', 'ols');
		
		rc = newdafne.Run();
		rc = newdafne.GetForecast ('Predict', weighted_forecast);

		/* 		nfor = newdafne.nfor(); */
		/* 		put nfor=; */
		rc = fitstats.Collect(newdafne);
		rc = forecast.Collect(newdafne);
	endsubmit;
run;

[GriloCanibal · Accepted solution]

Hi, Andrea. I'm here to help. I see some potential issues with the code you posted. For example, the following statements will assign missing values to the variables on the left-hand side of the assignment statement because the variables on the right-hand side are not defined anywhere in the program and, as a result, are automatically defined as numeric scalars carrying missing values:

p_fu[1] = P_Full;
ae_fu[1] = AE_Full;
p_ter[1] = P_Terna;
ae_ter[1] = AE_Terna;

Also, the program specifies the "outscalars p_fu p_ter;" statement in order to declare a set of scalar variables in the SAS program that will be copied to the OUTSCALAR= output CAS table. However, the SAS program then redefined those scalar variables as arrays. This could create issues. If you would like those variables to be used as arrays and also to be output to a CAS table, then you can move them from the "outscalars" statement to the "outarrays weighted_forecast;" statement. Furthermore, the program does not invoke the "TSM.Run()" method on the declared instances of the TSM object. As a result, no forecasts are produced and the CFC object cannot extract forecasts from the TSM object instances in order to create a combination forecast.

So that I may be able to help you with your programming task, would you kindly describe what data you have from the 6 NNET models? For example, the target variable (i.e., the original dependent series or Y-variable) is always required. To perform a combined forecast, you will also need the forecast series for each of the 6 NNET models. Note that if the "standard error" series of the 6 NNET forecasts are also available, then you can just combine the 6 NNET forecasts directly by adding them to a CFC object instance via the "CFC.AddPredict()" method and then invoking the "CFC.Run()" method to produce a combined forecast. However, if the standard error series of any of the 6 NNET forecasts are not available, then you will need to estimate the 'standard error" series by using the EXMSPEC/TSM objects. After initializing the TSM object with the EXMSPEC object and invoking the "TSM.Run()" method to produce a forecast and estimate the "standard error" series, you can then feed the resulting models to the CFC object via the "CFC.AddModel()" method. Finally, you will then invoke the "CFC.Run()" method to produce a combined forecast.

In summary, the correct approach to this problem depends on what type of data you have available from the 6 NNET model forecasts. If you can provide this information, then I should be able to provide better guidance on how to approach this problem. Thank you.

[andrea_magatti]

Thanks @GriloCanibal 

I'm responding now...

Better late than never.

Thanks for your suggestion. Everything works fine.

Just one more doubt:

after using the Run method, I'm able to extract the combined forecast, but is it possible to extract the weights used by the specific Criterion that has been used?

Thanks

[GriloCanibal]

Hi, Andrea. Unfortunately, the weights used in the combination forecast cannot be retrieved from the CFC object in the TSM package. However, if you have access to the Automated Time Series Modeling (ATSM) package, then you can also perform combination forecasts using that package and then retrieve the weights that were internally optimized for each member of the combination forecast via the OUTEST object, which stores the resulting model parameters into a CAS table. Please let me know if you have access to the ATSM package and I will look into providing you an example of how this can be achieved in that package. Thank you.

[andrea_magatti]

Thank you so much!

I have access to the atsm package.

I have read something like what you are talking about, and a simple example is really appreciated.

Thanks again!

[GriloCanibal]

Hi, Andrea. Please see below a sample code that uses the COMBSPEC and FORENG objects of the ATSM package to generate a combination forecast for the sample SASHELP.PREICEDATA data set using the best autogenerated ARIMA and ESM models. The parameters of the resulting combination model (that is, the combination weights of each model) are stored into the CAS table associated with the OUTEST object. This output table will also contain the parameters of the ARIMA and ESM models. Below is sample code that you can modify according to your needs:

/* Load the SASHELP.PRICEDATA set to CAS */
data sascas1.pricedata;
    set sashelp.pricedata;
run;

/* Invoke PROC TSMODEL to perform a forecast */
proc tsmodel data=sascas1.pricedata outlog=sascas1.outlog
    logcontrol=(error=keep warning=keep note=keep none=keep)
    outobj=(outfor=sascas1.outfor
            outest=sascas1.outest)
    lead=12 errorstop=yes puttolog=yes;
id date interval=month;
var sale price;
require atsm;
print outlog;
submit;
    /* Initialize the data frame with Y-series and independent variables */
    declare object dataFrame(tsdf);
    rc = dataFrame.initialize();
    rc = dataFrame.AddY(sale);
    rc = dataFrame.addX(price,'REQUIRED','YES');

    /* Specify the autogeneration of the best ARIMA and ESM models */
    declare object dspec(diagspec);
    rc = dspec.open();
    rc = dspec.SetARIMAX();
    rc = dspec.SetESM();
    rc = dspec.close();

    /* Run the model diagnostics to generate models */
    declare object diag(diagnose);
    rc = diag.initialize(dataFrame);
    rc = diag.setSpec(dspec);
    rc = diag.run();

    /* Initialize the combination list object to combine the ARIMA and ESM forecasts */
    declare object combo(combspec);
    rc = combo.open(diag.nmodels());
    rc = combo.addFrom(diag);
    rc = combo.setOption('WEIGHT','NERLS', 'MISSMODE','RESCALE');
    rc = combo.close();

    /* Perform the combination forecast */
    declare object forecast(foreng);
    rc = forecast.initialize(dataFrame);
    rc = forecast.addFrom(combo);
    rc = forecast.run();
    model = forecast.model();
    put model=;

    /* Store the forecast and model parameters (combination weights) into a CAS table */
    declare object outfor(outfor);
    declare object outest(outest);
    rc = outfor.collect(forecast);
    rc = outest.collect(forecast);
endsubmit;
quit;

/* Print the results */
proc print data=sascas1.outest;title 'OUTEST=';run;
proc print data=sascas1.outfor;title 'OUTFOR=';run;

Below are the contents of the resulting OUTEST table, where the combination weights for the ARIMA and ESM models are shown in bold red.

  Obs    _NAME_    _SELECT_    _MODEL_          _MODELVAR_    _DSVAR_    _VARTYPE_    _TRANSFORM_    _COMPONENT_      _COMPMODEL_

   1      sale     combo       DIAG1_ARIMAX1       Y           sale      DEPENDENT        NONE       MA
   2      sale     combo       DIAG1_ARIMAX1       Y           sale      DEPENDENT        NONE       AR
   3      sale     combo       DIAG1_ARIMAX1       Y           sale      DEPENDENT        NONE       AR
   4      sale     combo       DIAG1_ARIMAX1       price       price     INPUT            NONE       SCALE
   5      sale     combo       DIAG1_ESM1          Y           sale      DEPENDENT        NONE       LEVEL              SEASONAL
   6      sale     combo       DIAG1_ESM1          Y           sale      DEPENDENT        NONE       SEASON             SEASONAL
   7      sale     forecast    combo               Y           sale      DEPENDENT        NONE       DIAG1_ARIMAX1      NERLS
   8      sale     forecast    combo               Y           sale      DEPENDENT        NONE       DIAG1_ESM1         NERLS

  Obs    _PARM_    _FACTOR_    _LAG_    _SHIFT_           _EST_      _STDERR_        _TVALUE_        _PVALUE_

   1     MA1_12       1         12         0       0.6200602557    0.1436274752    4.3171423495     0.000091252
   2     AR1_1        1          1         0       -0.758154219    0.1308710553    -5.793139037    7.3050363E-7
   3     AR1_2        1          2         0       -0.547433876    0.1322246767    -4.140179346     0.000158864
   4     SCALE        0          0         0       -4.628375257    1.1159121362    -4.147616202    0.0001552294
   5                  1          0         0       0.0517660246    0.0188779895    2.7421365356    0.0081044568
   6                  1          0         0              0.001    0.0662298807     0.015098925    0.9880050778
   7     WEIGHT       0          0         0       0.3743681817    0.1587841476    2.3577176139    0.0226964096
   8     WEIGHT       0          0         0       0.6256318183    0.1587841476    3.9401402956    0.0002746819

  Obs    _LABEL_                                                                           _STATUS_

   1     ARIMA:  sale  ~ P = 2  D = (1,12)  Q = (12)   NOINT  +  INPUT: Dif(1,12) price       0
   2     ARIMA:  sale  ~ P = 2  D = (1,12)  Q = (12)   NOINT  +  INPUT: Dif(1,12) price       0
   3     ARIMA:  sale  ~ P = 2  D = (1,12)  Q = (12)   NOINT  +  INPUT: Dif(1,12) price       0
   4     ARIMA:  sale  ~ P = 2  D = (1,12)  Q = (12)   NOINT  +  INPUT: Dif(1,12) price       0
   5     Seasonal Exponential Smoothing                                                       0
   6     Seasonal Exponential Smoothing                                                       0
   7                                                                                          0
   8                                                                                          0

Please let me know of any other issues or questions.

[andrea_magatti]

Thank you so much!

I've used much of your code to adapt it to my specific case, which uses external forecast.

I'm posting my code for future users, in search of solutions!

options casdatalimit=ALL;
cas mysession sessopts=(caslib=casuser timeout=1800);
caslib _all_ assign;
caslib query datasource=(srctype="path") path="/sasdata/Stl_analysis/query/" 
	local;
libname query cas caslib=query;

proc cas;
	table.fileInfo result=fileList / kbytes=true;

	do row over fileList.FileInfo;
		print row.name;
		table.loadtable / path=row.name, caslib='query', casout={caslib='query', 
			name=substr(row.name , 1, index(row.name, '.')-1), replace=true};
	end;
quit;

/* test DS */
data casuser.test;
 set query.nn_rciv_g_preds_hist;
 keep date pred_: ae_: stl_tgt_ricciv_all P_rciv_ens;
 rename P_rciv_ens=prediction;
 rename stl_tgt_ricciv_all = target; 		
run;


proc tsmodel
	data=casuser.test outlog=casuser.outlog
    logcontrol=(error=keep warning=keep note=keep none=keep)
	outobj=(
			outfor=casuser.forecast
			outstat=casuser.stats
			outest =casuser.estimation
			);
	id date interval=day setmissing=missing;
	var ae_: pred_:;
	print outlog;

	require atsm ;
	require tsm;

	submit;
	    /* Initialize the data frame with Y-series and independent variables */
	    declare object dataFrame(tsdf);
	    rc = dataFrame.initialize();
	    rc = dataFrame.AddY(pred_globale_m1_1g);
	    rc = dataFrame.AddY(pred_globale_m1_2g);
	    rc = dataFrame.AddY(ae_globale_m1_1g);
	    rc = dataFrame.AddY(ae_globale_m1_2g);
	    rc = dataFrame.addX(date,'REQUIRED','YES');

		/* object for collecting results	*/
		declare object outstat(outstat);
		declare object outfor(outfor);
		declare object outest(outest);

		/* Specify external models to be imported		 */
		declare object ext_glob_m1_1g(exmspec);
		declare object ext_glob_m1_2g(exmspec);

		/* object responsible for combining models	*/
		declare object ensemble(COMBSPEC);

		/* object responsible for running forecast*/
		declare object makeForecast(FORENG);

		/* TSM object populated with external series*/
		declare object tsm_glob_m1_1g(tsm);
		declare object tsm_glob_m1_2g(tsm);

		/* for each external model we define the array*/
		/* containing Prediction and Absolute Error	*/
		rc = ext_glob_m1_1g.Open();
		rc = ext_glob_m1_1g.SetOption('Predict', 'pred_globale_m1_1g' ,'Error', 'ae_globale_m1_1g');
		rc = ext_glob_m1_1g.Close();
		rc = ext_glob_m1_2g.Open();
		rc = ext_glob_m1_2g.SetOption('Predict', 'pred_globale_m1_2g', 'Error', 'ae_globale_m1_2g');
		rc = ext_glob_m1_2g.Close();
		
		/* Initialization and run of first model */
		rc = tsm_glob_m1_1g.Initialize(ext_glob_m1_1g);
		rc = tsm_glob_m1_1g.AddExternal(pred_globale_m1_1g, 'predict');
		rc = tsm_glob_m1_1g.AddExternal(ae_globale_m1_1g, 'error');
		rc = tsm_glob_m1_1g.Run();

		/* Initialization and run of second model */
		rc = tsm_glob_m1_2g.Initialize(ext_glob_m1_2g);
		rc = tsm_glob_m1_2g.AddExternal(pred_globale_m1_2g, 'predict');
		rc = tsm_glob_m1_2g.AddExternal(ae_globale_m1_2g, 'error');
		rc = tsm_glob_m1_2g.Run();
		
		/* Combspec object preparation */
		rc=ensemble.Open(2);
		rc=ensemble.AddFrom(tsm_glob_m1_1g);
		rc=ensemble.AddFrom(tsm_glob_m1_2g);
		rc=ensemble.SetOption('WEIGHT', 'RANKWGT');
		rc=ensemble.Close();

		/* FORENG object execution */
		rc=makeforecast.initialize(dataFrame);	
		rc=makeForecast.addFrom(ensemble);
		rc=makeForecast.SetOption('CRITERION', 'MAE');
		rc=makeForecast.run();

		rc =outfor.collect(makeForecast, 'ALL');
		rc =outest.collect(makeForecast);
		rc =outest.collect(makeforecast);
	
	endsubmit;

quit;


/* Print the results */
proc print data=casuser.estimation;title 'OUTEST=';run;
proc print data=casuser.forecast;title 'OUTFOR=';run;

As you can see I've used object both from atsm package and from the tsm package to deal with external models ( Neural Net models)

Thanks again