Re: Convert Procedural DS2 code into MAS module and run it in Calculat...

jovic92 · Posted 10-12-2018 06:53 AM

I'm trying to migrate code from 4.3 to 5.2 but script doesn't work until I translate Procedural window DS2 code into MAS module.

Code I have in 4.3:

data esp.out;
        dcl integer ID;
        retain ID 0; 
        dcl double AGG3H;
        dcl integer EVENT_TYPE;
        dcl integer IS_TRIGGERED;
        dcl double TOTAL_BONUS;
        dcl double BONUS_LEFT;
        dcl timestamp LAST_USAGE_DTTM;
        dcl integer COUNT;
        dcl timestamp PREV_EXP_DTTM;
        dcl timestamp TO_BOUGHT_DTTM;
    method run();
        set esp.in;
            ID = ID + 1;
            BONUS_LEFT = -1 * RO_CCP1_USED_VOLUME;
            COUNT = 0;
            PREV_EXP_DTTM = to_timestamp(0); 
            TO_BOUGHT_DTTM = to_timestamp(2552317828);
            LAST_USAGE_DTTM = CALL_END_DTTM;
            TOTAL_BONUS = 0;
            IS_TRIGGERED = 0;
            EVENT_TYPE = 3;
            /*AGG3H = 0;*/
            _opcode = 1;
    end;
enddata;

And code I put in MAS module so I can call it in Calculate window:

ds2_options sas;
package module_1/overwrite=yes;
    method compute_usage(in_out int EVENT_TYPE, timestamp LAST_TO_BOUGHT_DTTM, in_out double BONUS_LEFT, in_out double TOTAL_BONUS, in_out timestamp EXP_DTTM, timestamp LAST_USAGE_DTTM, in_out timestamp PREV_EXP_DTTM, in_out double AVG_USAGE_LAST3HRS, in_out int COUNT, int TO_VALIDITY, int TO_BONUS);
        LAST_USAGE_DTTM = LAST_USAGE_DTTM;
        EXP_DTTM =to_timestamp(to_double(LAST_TO_BOUGHT_DTTM) + (TO_VALIDITY * 24 * 60 *60));
        BONUS_LEFT = TO_BONUS * 1024 * 1024;
        TOTAL_BONUS =  TO_BONUS * 1024 * 1024;
        AVG_USAGE_LAST3HRS = 0.0;
        COUNT = 1;
        EVENT_TYPE = 1;
        PREV_EXP_DTTM = TO_BOUGHT_DTTM;
    end;
endpackage;

But something doesn't work and logs are not really helpful in this case:

2018-10-12T11:59:58,383; DEBUG; 00004981; DF.ESP; (dfESPwindow_join.cpp:415); [Join0016] dfESPwindow_join::buildSchema() for window OneE_CDR_CM_To_Activation_Join: fieldName: MSISDN, side:1, type: 3, key: 0
2018-10-12T11:59:58,383; INFO ; 00004981; DF.ESP; (dfESPwindow_join.cpp:447); [Join0033] dfESPwindow_join::buildSchema() for window OneE_CDR_CM_To_Activation_Join: Built schema <ID*:int64,SUBSCRIPTION_SK:int64,TO_BOUGHT_DTTM:stamp,TO_BONUS:int64,TO_VALIDITY:int32,TO_KEYWORD:string,MSISDN:string>
2018-10-12T11:59:58,384; DEBUG; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Invoked calculate window finalizer
2018-10-12T11:59:58,384; DEBUG; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Successfully loaded tk extension "tksautil"
2018-10-12T11:59:58,386; DEBUG; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Successfully loaded tk extension "tksamas"
MAS map: window "OneE_CDR_CM_To_Activation_Join" -> (one_join,first_run,0)
2018-10-12T11:59:58,386; DEBUG; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Finished creating instance
2018-10-12T11:59:58,412; ERROR; 00004981; App.tk.MAS; (tksf.c:5313); Module 'one_join' failed to compile in user context '<esp>::<HDHA_migrate>'.
2018-10-12T11:59:58,412; INFO ; 00004981; DF.ESP.SA.TKSAMAS ; (sklstoj.c:118); Line 21: Invalid conversion for date or time type.
2018-10-12T11:59:58,412; ERROR; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Failed to initialize model
2018-10-12T11:59:58,412; ERROR; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Failed to finalize calculate window "<esp>::<HDHA_migrate>::<cq1>::<Calculate1>"
2018-10-12T11:59:58,412; ERROR; 00004981; DF.ESP; (dfESPwindow.cpp:1433); [Window0003] dfESPwindow::finalizeForRun() for window Calculate1: user user defined finalize callback returned failure
2018-10-12T11:59:58,413; ERROR; 00004981; DF.ESP; (dfESPwindow.cpp:1472); [Window0002] dfESPwindow::finalizeForRun() for window Calculate1: Finalization failed
2018-10-12T11:59:58,413; FATAL; 00004981; DF.ESP; (dfESPproject.cpp:640); [Project0004] dfESPproject::finalizeProject() for project HDHA_migrate: Failed finalizing continuous query: cq1
2018-10-12T11:59:58,413; DEBUG; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Invoked calculate window destructor

I was looking for some online examples, but all of them are pretty simple and basic. Any kind of help or hint is welcomed. Thanks in advance.

ballardw · Posted 10-12-2018 12:23 PM

@jovic92 wrote:

I'm trying to migrate code from 4.3 to 5.2 but script doesn't work until I translate Procedural window DS2 code into MAS module.

Code I have in 4.3:

data esp.out;
        dcl integer ID;
        retain ID 0; 
        dcl double AGG3H;
        dcl integer EVENT_TYPE;
        dcl integer IS_TRIGGERED;
        dcl double TOTAL_BONUS;
        dcl double BONUS_LEFT;
        dcl timestamp LAST_USAGE_DTTM;
        dcl integer COUNT;
        dcl timestamp PREV_EXP_DTTM;
        dcl timestamp TO_BOUGHT_DTTM;
    method run();
        set esp.in;
            ID = ID + 1;
            BONUS_LEFT = -1 * RO_CCP1_USED_VOLUME;
            COUNT = 0;
            PREV_EXP_DTTM = to_timestamp(0); 
            TO_BOUGHT_DTTM = to_timestamp(2552317828);
            LAST_USAGE_DTTM = CALL_END_DTTM;
            TOTAL_BONUS = 0;
            IS_TRIGGERED = 0;
            EVENT_TYPE = 3;
            /*AGG3H = 0;*/
            _opcode = 1;
    end;
enddata;

And code I put in MAS module so I can call it in Calculate window:

ds2_options sas;
package module_1/overwrite=yes;
    method compute_usage(in_out int EVENT_TYPE, timestamp LAST_TO_BOUGHT_DTTM, in_out double BONUS_LEFT, in_out double TOTAL_BONUS, in_out timestamp EXP_DTTM, timestamp LAST_USAGE_DTTM, in_out timestamp PREV_EXP_DTTM, in_out double AVG_USAGE_LAST3HRS, in_out int COUNT, int TO_VALIDITY, int TO_BONUS);
        LAST_USAGE_DTTM = LAST_USAGE_DTTM;
        EXP_DTTM =to_timestamp(to_double(LAST_TO_BOUGHT_DTTM) + (TO_VALIDITY * 24 * 60 *60));
        BONUS_LEFT = TO_BONUS * 1024 * 1024;
        TOTAL_BONUS =  TO_BONUS * 1024 * 1024;
        AVG_USAGE_LAST3HRS = 0.0;
        COUNT = 1;
        EVENT_TYPE = 1;
        PREV_EXP_DTTM = TO_BOUGHT_DTTM;
    end;
endpackage;

But something doesn't work and logs are not really helpful in this case:

2018-10-12T11:59:58,383; DEBUG; 00004981; DF.ESP; (dfESPwindow_join.cpp:415); [Join0016] dfESPwindow_join::buildSchema() for window OneE_CDR_CM_To_Activation_Join: fieldName: MSISDN, side:1, type: 3, key: 0
2018-10-12T11:59:58,383; INFO ; 00004981; DF.ESP; (dfESPwindow_join.cpp:447); [Join0033] dfESPwindow_join::buildSchema() for window OneE_CDR_CM_To_Activation_Join: Built schema <ID*:int64,SUBSCRIPTION_SK:int64,TO_BOUGHT_DTTM:stamp,TO_BONUS:int64,TO_VALIDITY:int32,TO_KEYWORD:string,MSISDN:string>
2018-10-12T11:59:58,384; DEBUG; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Invoked calculate window finalizer
2018-10-12T11:59:58,384; DEBUG; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Successfully loaded tk extension "tksautil"
2018-10-12T11:59:58,386; DEBUG; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Successfully loaded tk extension "tksamas"
MAS map: window "OneE_CDR_CM_To_Activation_Join" -> (one_join,first_run,0)
2018-10-12T11:59:58,386; DEBUG; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Finished creating instance
2018-10-12T11:59:58,412; ERROR; 00004981; App.tk.MAS; (tksf.c:5313); Module 'one_join' failed to compile in user context '<esp>::<HDHA_migrate>'.
2018-10-12T11:59:58,412; INFO ; 00004981; DF.ESP.SA.TKSAMAS ; (sklstoj.c:118); Line 21: Invalid conversion for date or time type.
2018-10-12T11:59:58,412; ERROR; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Failed to initialize model
2018-10-12T11:59:58,412; ERROR; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Failed to finalize calculate window "<esp>::<HDHA_migrate>::<cq1>::<Calculate1>"
2018-10-12T11:59:58,412; ERROR; 00004981; DF.ESP; (dfESPwindow.cpp:1433); [Window0003] dfESPwindow::finalizeForRun() for window Calculate1: user user defined finalize callback returned failure
2018-10-12T11:59:58,413; ERROR; 00004981; DF.ESP; (dfESPwindow.cpp:1472); [Window0002] dfESPwindow::finalizeForRun() for window Calculate1: Finalization failed
2018-10-12T11:59:58,413; FATAL; 00004981; DF.ESP; (dfESPproject.cpp:640); [Project0004] dfESPproject::finalizeProject() for project HDHA_migrate: Failed finalizing continuous query: cq1
2018-10-12T11:59:58,413; DEBUG; 00004981; DF.ESP.SA  ; (sklstoj.c:118); Invoked calculate window destructor

I was looking for some online examples, but all of them are pretty simple and basic. Any kind of help or hint is welcomed. Thanks in advance.

I think you might think carefully about this line:

2018-10-12T11:59:58,412; INFO ; 00004981; DF.ESP.SA.TKSAMAS ; (sklstoj.c:118); Line 21: Invalid conversion for date or time type.

and the values of any variables used in a date or time role.

AndyT_SAS · Posted 10-12-2018 01:03 PM

The timestamp type isn't supported as a DS2 data type by MAS. I think there is a debug message about this if you turn up the debug levels enough. Here is a table for the conversion between ESP data types and MAS datatypes for DS2 and Python

https://go.documentation.sas.com/?docsetId=masag&docsetTarget=n0nsznpd1r9wd8n1sd6qsm8efyjg.htm&docse...

You want the timestamp to be a bigint.

jovic92 · Posted 10-15-2018 08:50 AM

This means that I can't use a variable as a timestamp in a previous window from which I import part of schema?

If I use this

ds2_options sas;
package module_1/overwrite=yes;
    dcl int EVENT_TYPE;
    dcl double BONUS_LEFT;
    dcl double TOTAL_BONUS;
    dcl double AVG_USAGE_LAST3HRS;
    dcl BIGINT EXP_DTTM;
    dcl int COUNT;
    dcl int TO_VALIDITY;
    dcl int TO_BONUS;
    method first_run(in_out int EVENT_TYPE, timestamp LAST_TO_BOUGHT_DTTM, in_out double BONUS_LEFT, in_out double TOTAL_BONUS, in_out BIGINT EXP_DTTM, in_out BIGINT AVG_USAGE_LAST3HRS, in_out int COUNT, int TO_VALIDITY, int TO_BONUS);
        EXP_DTTM =(to_double(LAST_TO_BOUGHT_DTTM) + (TO_VALIDITY * 24 * 60 *60));
        BONUS_LEFT = TO_BONUS * 1024 * 1024;
        TOTAL_BONUS =  TO_BONUS * 1024 * 1024;
        AVG_USAGE_LAST3HRS = 0.0;
        COUNT = 1;
        EVENT_TYPE = 1;
    end;
endpackage;

Where LAST_TO_BOUGHT_DTTM is timestamp :

I only get

But if I put BIGINT instead timestamp for LAST_TO_BOUGHT_DTTM, I got an error like in question from the opening topic.

The previous window is "join window" and there are a few variables with timestamp type, how to handle that?

I couldn't find any example using BIGINT or any timestamp values, most time I run on volume = quantity * price and that's pretty basic.

Problem is that I can't start a new project (simplest one) until I migrate this one to 5.2 version.

AndyT_SAS · Posted 10-15-2018 03:29 PM

In ESP, a date or a timestamp is stored as a int64. A date is the number of seconds since Unix epoch. A timestamp is the number of milliseconds since Unix epoch. So, it should be able to be passed as a bigint to DS2, which will correctly hold the value. I'm not sure why the model won't start when LAST_TO_BOUGHT_DTTM is set to bigint. Can you add the model so I can take a look? Or we can create a track for this, if you would like?

jovic92 · Posted 10-15-2018 04:13 PM

Andy,

I would like if you can create a track for this so I can send you a mail with the model in the attachment.

Thanks,

Best regards

Convert Procedural DS2 code into MAS module and run it in Calculate window

Re: Convert Procedural DS2 code into MAS module and run it in Calculate window

Re: Convert Procedural DS2 code into MAS module and run it in Calculate window

Re: Convert Procedural DS2 code into MAS module and run it in Calculate window

Re: Convert Procedural DS2 code into MAS module and run it in Calculate window

Re: Convert Procedural DS2 code into MAS module and run it in Calculate window