SAS ESP - Streaming of data is slow

[upadhi]

Hi all,

I am using ESP for the first time. below is the command  I am using to fetch data from CSV file:

$DFESP_HOME/bin/dfesp_fs_adapter -k pub -h "dfESP://sasserver.demo.sas.com:8701/SGF_Test_V2/sgf_cq/sgf_source" -f "/home/sas/data2_root.csv" -t csv -b 1 -r 1

I am using ESP 5.2

The issue is that the streaming of data is happening very slowly. What are the parameter /configurations that could be changed to get a faster stream of data.

Thanks and regards,

Upadhi

[FredCombaneyre]

Hi, 

just remove the -r 1 option. it sets the input rate to 1 event per second. 

[upadhi]

hi Fred, If i remove -r, all the records come in once and i can not see stream of data incoming. Thats why i kept -r to 1

[FredCombaneyre]

They cannot be coming in once, even more that you force the block size to 1 (-b 1). This is just the refresh delay of the tool you are using for displaying the events that makes it look as such. 

[ScottK · Accepted solution]

If you remove the -r 1, but leave the -b 1, they will indeed be streamed in 1 at a time. If you only have a few rows, it will appear that they are all coming in at once, since ESP an handle multiple thousands of csv events per second. if you have only a small data set, experiment with -r N for various values of N. N is the target rate the adapter tries to maintain. For slow rates (< 1000) it is more approximate. If you have 1000 rows, and set -r 100, it should stream data for approximates 10 seconds. 

Hope that helps, 

Scott.