07-11-2024
AndyT_SAS
SAS Employee
Member since
11-07-2014
- 25 Posts
- 3 Likes Given
- 2 Solutions
- 8 Likes Received
-
Latest posts by AndyT_SAS
Subject Views Posted 1628 01-31-2024 11:58 AM 1537 03-27-2019 07:40 AM 1560 03-26-2019 02:29 PM 853 03-14-2019 10:00 AM 1252 01-21-2019 09:33 AM 1297 01-15-2019 12:53 PM 1125 01-03-2019 11:39 AM 1069 11-09-2018 07:02 AM 2412 10-15-2018 03:29 PM 2457 10-12-2018 01:03 PM -
Activity Feed for AndyT_SAS
- Liked Re: Authentication when using the URL_Connector to query data from a REST API for VMoraal. 02-01-2024 11:37 AM
- Posted Re: Authentication when using the URL_Connector to query data from a REST API on Streaming Analytics. 01-31-2024 11:58 AM
- Liked 2024 Customer Awards: Georgia-Pacific - Curious Thinker for slcoyne86. 01-31-2024 11:37 AM
- Got a Like for Re: I want to connect OPC UA Server Using SAS ESP. How to establish connection ?. 04-14-2020 11:08 AM
- Posted Re: SAS ESP 5.2 Oracle pub/sub connector fails to start on Streaming Analytics. 03-27-2019 07:40 AM
- Posted Re: SAS ESP 5.2 Oracle pub/sub connector fails to start on Streaming Analytics. 03-26-2019 02:29 PM
- Posted Re: How can I stream Twitter data in SAS ESP 5.2 on Streaming Analytics. 03-14-2019 10:00 AM
- Posted Re: removing Opcode and eventflag chars in CSV output on Streaming Analytics. 01-21-2019 09:33 AM
- Posted Re: removing Opcode and eventflag chars in CSV output on Streaming Analytics. 01-15-2019 12:53 PM
- Posted Re: Discarding duplicate events produced by QKB matchcode function on Streaming Analytics. 01-03-2019 11:39 AM
- Got a Like for Re: What would be the best approach to handle offline situations for ESP on edge devices?. 11-13-2018 01:19 PM
- Posted Re: What would be the best approach to handle offline situations for ESP on edge devices? on Streaming Analytics. 11-09-2018 07:02 AM
- Posted Re: Convert Procedural DS2 code into MAS module and run it in Calculate window on Streaming Analytics. 10-15-2018 03:29 PM
- Posted Re: Convert Procedural DS2 code into MAS module and run it in Calculate window on Streaming Analytics. 10-12-2018 01:03 PM
- Posted Re: Hex values in ESP on Streaming Analytics. 10-10-2018 12:16 PM
- Posted Re: SAS ESP reading data from Apache NiFi on Streaming Analytics. 10-09-2018 01:00 PM
- Got a Like for Re: SAS ESP reading data from Apache NiFi. 10-09-2018 11:25 AM
- Got a Like for Re: SAS ESP reading data from Apache NiFi. 10-09-2018 10:46 AM
- Posted Re: SAS ESP reading data from Apache NiFi on Streaming Analytics. 10-09-2018 10:31 AM
- Posted Re: SAS ESP reading data from Apache NiFi on Streaming Analytics. 10-05-2018 10:28 AM
-
Posts I Liked
Subject Likes Author Latest Post 1 75 -
My Liked Posts
Subject Likes Posted 1 08-09-2018 02:17 PM 1 11-09-2018 07:02 AM 2 10-09-2018 10:31 AM 1 08-15-2018 12:50 PM 3 03-01-2018 09:40 AM -
My Library Contributions
Subject Likes Author Latest Post 0 1 2
01-31-2024
11:58 AM
I don't think the url connector is expecting user/password information in the url. I will submit a request to update ESP to support that. What release of ESP are you using? That will help us find a work around.
... View more
03-27-2019
07:40 AM
ESP Studio is reflecting the message from the ESP XML server about the connector not starting. Can you try running without the -logconfig option to see if there are any additional messages about the connector not starting?
... View more
03-26-2019
02:29 PM
There are normally error messages that go with the "connector failed to start". The DB connector/adapter will try to connect to the database as part of the start up process.
This could be because of how the logging is set up. What do you have configured in the esp.logger file?
... View more
03-14-2019
10:00 AM
You can use the Twitter Publisher Adapter to publish tweets into an ESP model. Here is a blog post that shows more details on using the twitter adapter and the ESP model that can be designed to process the tweets.
... View more
01-21-2019
09:33 AM
Hi Ebele,
The file/socket connector does not allow an option to remove opcode or the flags from the format. What is the intended target of the events? Who will process the data? Is the model only going to output insert events?
Thanks,
Andy
... View more
01-15-2019
12:53 PM
Hi Ebele,
There is not a way to remove the opcode or event flag from the CSV output. The format was designed so that it could be read back in by another ESP source window.
Thanks,
Andy
... View more
01-03-2019
11:39 AM
Hi Rain,
This is a little tricky. How many match codes could you potentially get? An aggregate window could group events by the matchcode. But you would need to worry about storage growth in the aggregate window because nothing will remove the old events.
.
You could write a Python or DS2 routine with a hash table that uses the matchcode has the hash key. Then, you could keep a record of the matchcode values that have been processed. An event would only be output if it was a new matchcode. That wouldn't have any memory concerns, as you could make the window pi_EMPTY.
Thanks,
Andy
... View more
11-09-2018
07:02 AM
1 Like
Hi Ebele,
I see you opened a track for this. We will work with you on the track, then update this with the solution.
Thanks,
Andy
... View more
10-15-2018
03:29 PM
In ESP, a date or a timestamp is stored as a int64. A date is the number of seconds since Unix epoch. A timestamp is the number of milliseconds since Unix epoch. So, it should be able to be passed as a bigint to DS2, which will correctly hold the value. I'm not sure why the model won't start when LAST_TO_BOUGHT_DTTM is set to bigint. Can you add the model so I can take a look? Or we can create a track for this, if you would like?
... View more
10-12-2018
01:03 PM
The timestamp type isn't supported as a DS2 data type by MAS. I think there is a debug message about this if you turn up the debug levels enough. Here is a table for the conversion between ESP data types and MAS datatypes for DS2 and Python
https://go.documentation.sas.com/?docsetId=masag&docsetTarget=n0nsznpd1r9wd8n1sd6qsm8efyjg.htm&docsetVersion=5.1&locale=en
You want the timestamp to be a bigint.
... View more
10-10-2018
12:16 PM
There isn't a built in function for this in the functional window or compute window. You could use a python window or write a C++ function to do this.
... View more
10-09-2018
01:00 PM
I would open a new question. That way the discussions don't get confusing for someone else who finds this thread in the future.
... View more
10-09-2018
10:31 AM
2 Likes
Hi Sinisa,
I'm not sure what is happening here. I'm going to create a track for you and we can investigate it from there. You will get an email from the track.
Thanks,
Andy
... View more
10-05-2018
10:28 AM
You can get the AVRO format of an ESP window's schema using this REST request:
http://hostname:http_port/SASESP/window?name=xxxxx&schema=true&avro=true
Substitute the window name you are trying to publish to for xxxxx.
But if the Record Schema isn't specified, ESP should connect to the running project and obtain the schema for you. Is the ESP project up when you try to start the processor, it should connect ok. What error message do you get?
... View more
09-10-2018
10:25 AM
In Event Stream Processing(ESP) 5.2, support for the Procedural window input handler that used DS2 code run in table server mode is deprecated. This means that to migrate from previous releases of Event Stream Processing to release 5.2, the DS2 table server mode code must be changed to run as a MAS module. The MAS module then needs to be moved to the calculate window. The following code snippet changes the opcode of incoming events. Here’s how to convert it from using DS2 to MAS.
Here is the existing procedural window in ESP 5.1:
<window-procedural pubsub="true" output-insert-only="true" index="pi_EMPTY" produces-only-inserts="true" pubsub-index="pi_EMPTY" name="Procedural1"> <schema> <fields> <field name="id" type="int64" key="true"/> <field name="sum" type="double"/> <field name="average" type="double"/> </fields> </schema> <ds2-tableserver source="input_window"> <code><![CDATA[ds2_options cdump; data esp.out; method run(); set esp.in; if _opcode = 3 then return; else do; _opcode = 1; end; end; enddata;]]></code> </ds2-tableserver> </window-procedural>
This code looks at the _opcode value. For _opcode of 3, which is a delete, the event is dropped. For all other _opcodes, the _opcode is set to 1, which is an insert. This module does not modify or use any of the fields in the input schema or output schema of the model. This means there are no field values to pass to the MAS module. The MAS module will need the input opcode and the output opcode. The opcode values used by MAS are strings instead of integers. There is a conversion table for the opcodes in the Event Stream Processing 5.2 User's Guide, which also documents additional information on migrating to a MAS module.
Here is the MAS module DS2 code to convert the opcodes:
<mas-modules> <mas-module language="ds2" module="module_1" func-names="Make_Insert"> <code> ds2_options sas; package module_1/overwrite=yes; /* Declare the function. Must declare the input opcode and the output opcode */ method Make_Insert(varchar(16) _inOpcode, in_out varchar _outOpcode); if _inOpcode = 'delete' then do; _outOpcode = 'noop'; /* Suppress event */ end; else do; _outOpcode = 'insert'; /* Set the output event to be an Insert */ end; /* End Make_Insert method */ endpackage; </code> </mas-module> </mas-modules>
The function defines a method(Make_Insert). The method has two parameters, the input opcode(_inOpcode) and the output opcode(_outOpcode). The in_out definition tells MAS that _outOpcode is an output variable. The rest of the code is very similar to the previous function with the integer opcode values replaced with strings and the correct opcode variable used. The special value "noop" is used to tell ESP not to output an event. With this new MAS module, the procedural window can be changed to:
<window-procedural pubsub="true" output-insert-only="true" index="pi_EMPTY" produces-only-inserts="true" pubsub-index="pi_EMPTY" name="Procedural1"> <schema> <fields> <field name="id" type="int64" key="true"/> <field name="sum" type="double"/> <field name="average" type="double"/> </fields> </schema> <mas-map> <window-map module="module_1" function="Make_Insert" revision="0" source="input_window"/> </mas-map> </window-procedural>
The window-map tag refers back to the mas-module previously defined. None of the schema fields are changed. Now that the procedural window is updated to use the MAS module, the final piece of the conversion to 5.2 is done with the dfesp_xml_migrate tool. The dfesp_xml_migrate tool will convert the procedural window to a calculate window in the XML model. The XML model can now be run on ESP 5.2.
... View more
Labels: