Named Pipes — Easy Dynamic Integration for SAS ESP

[#1]

If you asked me to name an underutilized data integration technology, I might say named pipes. They may not be fancy but they are a powerful mechanism for integrating processes. Too often, I see process flows where one job waits for a predecessor job to completely finish before it can start. In many circumstances, named pipes could alleviate these bottlenecks.

If you're not familiar with named pipes, the important thing to know is that they allow two I/O dependent processes (one reading the output of the other) to run simultaneously without the need for any disk writes.  For example, if a SAS program writes a file that feeds a Python program, you can replace the file with a named pipe and the Python progam can read the pipe as the SAS program creates it.  The Python program doesn't have to wait for the SAS program to finish writing the file, and, since pipes are basically memory buffers, there is no need to write anything to disk.

Pipes have some stringent buffer limitations but these only only matter if the process consuming the data through the pipe cannot keep pace with the process loading the pipe. So, in many messaging and event-stream applications, named pipes offer an easy integration mechanism.

Let's look at a simple example of SAS ESP loading a named pipe, passing data to some down-stream process. The first thing we need to do is create our pipe. This will run on Linux so we'll use the mkfifo command.

mkfifo -m 766 myPipe

With the pipe created, I'll now run a program to read the pipe. We could use SAS, Python, Perl, or any other program that can consume a file. Here I'll use a simple cat command to minimize the complexity of the exercise.

cat myPipe

Now let's configure a SAS ESP project to write to the pipe. Since it's just for demonstration, I'll use any existing ESP project and add a subscription connector to write to myPipe.  (I'll run the ESP server in the same location as myPipe so no path is required.)

Now we run the project and look at the model output streaming to our cat command. The screen capture below shows the events, trade transactions, flowing through the cat program.

So there you have it, an easy, real-time, zero disk read/write technique for integrating ESP with 3rd party applications.  Of course, if you need more control, tighter integration, and/or even lower overhead, you can always use the ESP Pub/Sub API. .