topic SAS ESP - Streaming of data is slow in Streaming Analytics

SAS ESP - Streaming of data is slow

upadhi — Mon, 18 Mar 2019 05:40:57 GMT

Hi all,

I am using ESP for the first time. below is the command I am using to fetch data from CSV file:

$DFESP_HOME/bin/dfesp_fs_adapter -k pub -h "dfESP://sasserver.demo.sas.com:8701/SGF_Test_V2/sgf_cq/sgf_source" -f "/home/sas/data2_root.csv" -t csv -b 1 -r 1

I am using ESP 5.2

The issue is that the streaming of data is happening very slowly. What are the parameter /configurations that could be changed to get a faster stream of data.

Thanks and regards,

Upadhi

Re: SAS ESP - Streaming of data is slow

FredCombaneyre — Mon, 18 Mar 2019 07:46:17 GMT

Hi,

just remove the -r 1 option. it sets the input rate to 1 event per second.

Re: SAS ESP - Streaming of data is slow

upadhi — Mon, 18 Mar 2019 08:07:17 GMT

hi Fred, If i remove -r, all the records come in once and i can not see stream of data incoming. Thats why i kept -r to 1

Re: SAS ESP - Streaming of data is slow

FredCombaneyre — Mon, 18 Mar 2019 08:41:57 GMT

They cannot be coming in once, even more that you force the block size to 1 (-b 1). This is just the refresh delay of the tool you are using for displaying the events that makes it look as such.

Re: SAS ESP - Streaming of data is slow

ScottK — Mon, 18 Mar 2019 19:40:21 GMT

If you remove the -r 1, but leave the -b 1, they will indeed be streamed in 1 at a time. If you only have a few rows, it will appear that they are all coming in at once, since ESP an handle multiple thousands of csv events per second. if you have only a small data set, experiment with -r N for various values of N. N is the target rate the adapter tries to maintain. For slow rates (< 1000) it is more approximate. If you have 1000 rows, and set -r 100, it should stream data for approximates 10 seconds.

Hope that helps,

Scott.