topic Convert Procedural DS2 code into MAS module and run it in Calculate window in Streaming Analytics

Convert Procedural DS2 code into MAS module and run it in Calculate window

jovic92 — Fri, 12 Oct 2018 10:53:36 GMT

I'm trying to migrate code from 4.3 to 5.2 but script doesn't work until I translate Procedural window DS2 code into MAS module.

Code I have in 4.3:

data esp.out;
        dcl integer ID;
        retain ID 0; 
        dcl double AGG3H;
        dcl integer EVENT_TYPE;
        dcl integer IS_TRIGGERED;
        dcl double TOTAL_BONUS;
        dcl double BONUS_LEFT;
        dcl timestamp LAST_USAGE_DTTM;
        dcl integer COUNT;
        dcl timestamp PREV_EXP_DTTM;
        dcl timestamp TO_BOUGHT_DTTM;
    method run();
        set esp.in;
            ID = ID + 1;
            BONUS_LEFT = -1 * RO_CCP1_USED_VOLUME;
            COUNT = 0;
            PREV_EXP_DTTM = to_timestamp(0); 
            TO_BOUGHT_DTTM = to_timestamp(2552317828);
            LAST_USAGE_DTTM = CALL_END_DTTM;
            TOTAL_BONUS = 0;
            IS_TRIGGERED = 0;
            EVENT_TYPE = 3;
            /*AGG3H = 0;*/
            _opcode = 1;
    end;
enddata;

And code I put in MAS module so I can call it in Calculate window:

ds2_options sas;
package module_1/overwrite=yes;
    method compute_usage(in_out int EVENT_TYPE, timestamp LAST_TO_BOUGHT_DTTM, in_out double BONUS_LEFT, in_out double TOTAL_BONUS, in_out timestamp EXP_DTTM, timestamp LAST_USAGE_DTTM, in_out timestamp PREV_EXP_DTTM, in_out double AVG_USAGE_LAST3HRS, in_out int COUNT, int TO_VALIDITY, int TO_BONUS);
        LAST_USAGE_DTTM = LAST_USAGE_DTTM;
        EXP_DTTM =to_timestamp(to_double(LAST_TO_BOUGHT_DTTM) + (TO_VALIDITY * 24 * 60 *60));
        BONUS_LEFT = TO_BONUS * 1024 * 1024;
        TOTAL_BONUS =  TO_BONUS * 1024 * 1024;
        AVG_USAGE_LAST3HRS = 0.0;
        COUNT = 1;
        EVENT_TYPE = 1;
        PREV_EXP_DTTM = TO_BOUGHT_DTTM;
    end;
endpackage;

But something doesn't work and logs are not really helpful in this case:

2018-10-12T11:59:58,383; DEBUG; 00004981; DF.ESP; (dfESPwindow_join.cpp:415); [Join0016] dfESPwindow_join::buildSchema() for window OneE_CDR_CM_To_Activation_Join: fieldName: MSISDN, side:1, type: 3, key: 0
2018-10-12T11:59:58,383; INFO ; 00004981; DF.ESP; (dfESPwindow_join.cpp:447); [Join0033] dfESPwindow_join::buildSchema() for window OneE_CDR_CM_To_Activation_Join: Built schema <ID*:int64,SUBSCRIPTION_SK:int64,TO_BOUGHT_DTTM:stamp,TO_BONUS:int64,TO_VALIDITY:int32,TO_KEYWORD:string,MSISDN:string>
2018-10-12T11:59:58,384; DEBUG; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Invoked calculate window finalizer
2018-10-12T11:59:58,384; DEBUG; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Successfully loaded tk extension "tksautil"
2018-10-12T11:59:58,386; DEBUG; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Successfully loaded tk extension "tksamas"
MAS map: window "OneE_CDR_CM_To_Activation_Join" -> (one_join,first_run,0)
2018-10-12T11:59:58,386; DEBUG; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Finished creating instance
2018-10-12T11:59:58,412; ERROR; 00004981; App.tk.MAS; (tksf.c:5313); Module 'one_join' failed to compile in user context '<esp>::<HDHA_migrate>'.
2018-10-12T11:59:58,412; INFO ; 00004981; DF.ESP.SA.TKSAMAS ; (sklstoj.c:118); Line 21: Invalid conversion for date or time type.
2018-10-12T11:59:58,412; ERROR; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Failed to initialize model
2018-10-12T11:59:58,412; ERROR; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Failed to finalize calculate window "<esp>::<HDHA_migrate>::<cq1>::<Calculate1>"
2018-10-12T11:59:58,412; ERROR; 00004981; DF.ESP; (dfESPwindow.cpp:1433); [Window0003] dfESPwindow::finalizeForRun() for window Calculate1: user user defined finalize callback returned failure
2018-10-12T11:59:58,413; ERROR; 00004981; DF.ESP; (dfESPwindow.cpp:1472); [Window0002] dfESPwindow::finalizeForRun() for window Calculate1: Finalization failed
2018-10-12T11:59:58,413; FATAL; 00004981; DF.ESP; (dfESPproject.cpp:640); [Project0004] dfESPproject::finalizeProject() for project HDHA_migrate: Failed finalizing continuous query: cq1
2018-10-12T11:59:58,413; DEBUG; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Invoked calculate window destructor

I was looking for some online examples, but all of them are pretty simple and basic. Any kind of help or hint is welcomed. Thanks in advance.

Re: Convert Procedural DS2 code into MAS module and run it in Calculate window

ballardw — Fri, 12 Oct 2018 16:23:49 GMT

@jovic92 wrote:

I'm trying to migrate code from 4.3 to 5.2 but script doesn't work until I translate Procedural window DS2 code into MAS module.

Code I have in 4.3:

data esp.out;
        dcl integer ID;
        retain ID 0; 
        dcl double AGG3H;
        dcl integer EVENT_TYPE;
        dcl integer IS_TRIGGERED;
        dcl double TOTAL_BONUS;
        dcl double BONUS_LEFT;
        dcl timestamp LAST_USAGE_DTTM;
        dcl integer COUNT;
        dcl timestamp PREV_EXP_DTTM;
        dcl timestamp TO_BOUGHT_DTTM;
    method run();
        set esp.in;
            ID = ID + 1;
            BONUS_LEFT = -1 * RO_CCP1_USED_VOLUME;
            COUNT = 0;
            PREV_EXP_DTTM = to_timestamp(0); 
            TO_BOUGHT_DTTM = to_timestamp(2552317828);
            LAST_USAGE_DTTM = CALL_END_DTTM;
            TOTAL_BONUS = 0;
            IS_TRIGGERED = 0;
            EVENT_TYPE = 3;
            /*AGG3H = 0;*/
            _opcode = 1;
    end;
enddata;

And code I put in MAS module so I can call it in Calculate window:

ds2_options sas;
package module_1/overwrite=yes;
    method compute_usage(in_out int EVENT_TYPE, timestamp LAST_TO_BOUGHT_DTTM, in_out double BONUS_LEFT, in_out double TOTAL_BONUS, in_out timestamp EXP_DTTM, timestamp LAST_USAGE_DTTM, in_out timestamp PREV_EXP_DTTM, in_out double AVG_USAGE_LAST3HRS, in_out int COUNT, int TO_VALIDITY, int TO_BONUS);
        LAST_USAGE_DTTM = LAST_USAGE_DTTM;
        EXP_DTTM =to_timestamp(to_double(LAST_TO_BOUGHT_DTTM) + (TO_VALIDITY * 24 * 60 *60));
        BONUS_LEFT = TO_BONUS * 1024 * 1024;
        TOTAL_BONUS =  TO_BONUS * 1024 * 1024;
        AVG_USAGE_LAST3HRS = 0.0;
        COUNT = 1;
        EVENT_TYPE = 1;
        PREV_EXP_DTTM = TO_BOUGHT_DTTM;
    end;
endpackage;

But something doesn't work and logs are not really helpful in this case:

2018-10-12T11:59:58,383; DEBUG; 00004981; DF.ESP; (dfESPwindow_join.cpp:415); [Join0016] dfESPwindow_join::buildSchema() for window OneE_CDR_CM_To_Activation_Join: fieldName: MSISDN, side:1, type: 3, key: 0
2018-10-12T11:59:58,383; INFO ; 00004981; DF.ESP; (dfESPwindow_join.cpp:447); [Join0033] dfESPwindow_join::buildSchema() for window OneE_CDR_CM_To_Activation_Join: Built schema <ID*:int64,SUBSCRIPTION_SK:int64,TO_BOUGHT_DTTM:stamp,TO_BONUS:int64,TO_VALIDITY:int32,TO_KEYWORD:string,MSISDN:string>
2018-10-12T11:59:58,384; DEBUG; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Invoked calculate window finalizer
2018-10-12T11:59:58,384; DEBUG; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Successfully loaded tk extension "tksautil"
2018-10-12T11:59:58,386; DEBUG; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Successfully loaded tk extension "tksamas"
MAS map: window "OneE_CDR_CM_To_Activation_Join" -> (one_join,first_run,0)
2018-10-12T11:59:58,386; DEBUG; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Finished creating instance
2018-10-12T11:59:58,412; ERROR; 00004981; App.tk.MAS; (tksf.c:5313); Module 'one_join' failed to compile in user context '<esp>::<HDHA_migrate>'.
2018-10-12T11:59:58,412; INFO ; 00004981; DF.ESP.SA.TKSAMAS ; (sklstoj.c:118); Line 21: Invalid conversion for date or time type.
2018-10-12T11:59:58,412; ERROR; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Failed to initialize model
2018-10-12T11:59:58,412; ERROR; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Failed to finalize calculate window "<esp>::<HDHA_migrate>::<cq1>::<Calculate1>"
2018-10-12T11:59:58,412; ERROR; 00004981; DF.ESP; (dfESPwindow.cpp:1433); [Window0003] dfESPwindow::finalizeForRun() for window Calculate1: user user defined finalize callback returned failure
2018-10-12T11:59:58,413; ERROR; 00004981; DF.ESP; (dfESPwindow.cpp:1472); [Window0002] dfESPwindow::finalizeForRun() for window Calculate1: Finalization failed
2018-10-12T11:59:58,413; FATAL; 00004981; DF.ESP; (dfESPproject.cpp:640); [Project0004] dfESPproject::finalizeProject() for project HDHA_migrate: Failed finalizing continuous query: cq1
2018-10-12T11:59:58,413; DEBUG; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Invoked calculate window destructor

I was looking for some online examples, but all of them are pretty simple and basic. Any kind of help or hint is welcomed. Thanks in advance.

I think you might think carefully about this line:

2018-10-12T11:59:58,412; INFO ; 00004981; DF.ESP.SA.TKSAMAS ; (sklstoj.c:118); Line 21: Invalid conversion for date or time type.

and the values of any variables used in a date or time role.

Re: Convert Procedural DS2 code into MAS module and run it in Calculate window

AndyT_SAS — Fri, 12 Oct 2018 17:03:27 GMT

The timestamp type isn't supported as a DS2 data type by MAS. I think there is a debug message about this if you turn up the debug levels enough. Here is a table for the conversion between ESP data types and MAS datatypes for DS2 and Python

https://go.documentation.sas.com/?docsetId=masag&docsetTarget=n0nsznpd1r9wd8n1sd6qsm8efyjg.htm&docsetVersion=5.1&locale=en

You want the timestamp to be a bigint.

Re: Convert Procedural DS2 code into MAS module and run it in Calculate window

jovic92 — Mon, 15 Oct 2018 12:50:20 GMT

This means that I can't use a variable as a timestamp in a previous window from which I import part of schema?

If I use this

ds2_options sas;
package module_1/overwrite=yes;
    dcl int EVENT_TYPE;
    dcl double BONUS_LEFT;
    dcl double TOTAL_BONUS;
    dcl double AVG_USAGE_LAST3HRS;
    dcl BIGINT EXP_DTTM;
    dcl int COUNT;
    dcl int TO_VALIDITY;
    dcl int TO_BONUS;
    method first_run(in_out int EVENT_TYPE, timestamp LAST_TO_BOUGHT_DTTM, in_out double BONUS_LEFT, in_out double TOTAL_BONUS, in_out BIGINT EXP_DTTM, in_out BIGINT AVG_USAGE_LAST3HRS, in_out int COUNT, int TO_VALIDITY, int TO_BONUS);
        EXP_DTTM =(to_double(LAST_TO_BOUGHT_DTTM) + (TO_VALIDITY * 24 * 60 *60));
        BONUS_LEFT = TO_BONUS * 1024 * 1024;
        TOTAL_BONUS =  TO_BONUS * 1024 * 1024;
        AVG_USAGE_LAST3HRS = 0.0;
        COUNT = 1;
        EVENT_TYPE = 1;
    end;
endpackage;

Where LAST_TO_BOUGHT_DTTM is timestamp :

I only get

But if I put BIGINT instead timestamp for LAST_TO_BOUGHT_DTTM, I got an error like in question from the opening topic.

The previous window is "join window" and there are a few variables with timestamp type, how to handle that?

I couldn't find any example using BIGINT or any timestamp values, most time I run on volume = quantity * price and that's pretty basic.

Problem is that I can't start a new project (simplest one) until I migrate this one to 5.2 version.

Re: Convert Procedural DS2 code into MAS module and run it in Calculate window

AndyT_SAS — Mon, 15 Oct 2018 19:29:49 GMT

In ESP, a date or a timestamp is stored as a int64. A date is the number of seconds since Unix epoch. A timestamp is the number of milliseconds since Unix epoch. So, it should be able to be passed as a bigint to DS2, which will correctly hold the value. I'm not sure why the model won't start when LAST_TO_BOUGHT_DTTM is set to bigint. Can you add the model so I can take a look? Or we can create a track for this, if you would like?

Re: Convert Procedural DS2 code into MAS module and run it in Calculate window

jovic92 — Mon, 15 Oct 2018 20:14:00 GMT

Andy,

I would like if you can create a track for this so I can send you a mail with the model in the attachment.

Thanks,

Best regards