BookmarkSubscribeRSS Feed

Named Pipes — Easy Dynamic Integration for SAS ESP

Started ‎03-22-2017 by
Modified ‎03-29-2017 by
Views 1,688

If you asked me to name an underutilized data integration technology, I might say named pipes. They may not be fancy but they are a powerful mechanism for integrating processes. Too often, I see process flows where one job waits for a predecessor job to completely finish before it can start. In many circumstances, named pipes could alleviate these bottlenecks.

 

If you're not familiar with named pipes, the important thing to know is that they allow two I/O dependent processes (one reading the output of the other) to run simultaneously without the need for any disk writes.  For example, if a SAS program writes a file that feeds a Python program, you can replace the file with a named pipe and the Python progam can read the pipe as the SAS program creates it.  The Python program doesn't have to wait for the SAS program to finish writing the file, and, since pipes are basically memory buffers, there is no need to write anything to disk.

 

Pipes have some stringent buffer limitations but these only only matter if the process consuming the data through the pipe cannot keep pace with the process loading the pipe. So, in many messaging and event-stream applications, named pipes offer an easy integration mechanism.

 

Let's look at a simple example of SAS ESP loading a named pipe, passing data to some down-stream process. The first thing we need to do is create our pipe. This will run on Linux so we'll use the mkfifo command.

mkfifo -m 766 myPipe

With the pipe created, I'll now run a program to read the pipe. We could use SAS, Python, Perl, or any other program that can consume a file. Here I'll use a simple cat command to minimize the complexity of the exercise.

cat myPipe

Now let's configure a SAS ESP project to write to the pipe. Since it's just for demonstration, I'll use any existing ESP project and add a subscription connector to write to myPipe.  (I'll run the ESP server in the same location as myPipe so no path is required.)

 

pipeConnector.png

 

Now we run the project and look at the model output streaming to our cat command. The screen capture below shows the events, trade transactions, flowing through the cat program.

 

catOutput.png

 

So there you have it, an easy, real-time, zero disk read/write technique for integrating ESP with 3rd party applications.  Of course, if you need more control, tighter integration, and/or even lower overhead, you can always use the ESP Pub/Sub API. .

Version history
Last update:
‎03-29-2017 03:00 PM
Updated by:
Contributors

hackathon24-white-horiz.png

The 2025 SAS Hackathon Kicks Off on June 11!

Watch the live Hackathon Kickoff to get all the essential information about the SAS Hackathon—including how to join, how to participate, and expert tips for success.

YouTube LinkedIn

SAS AI and Machine Learning Courses

The rapid growth of AI technologies is driving an AI skills gap and demand for AI talent. Ready to grow your AI literacy? SAS offers free ways to get started for beginners, business leaders, and analytics professionals of all skill levels. Your future self will thank you.

Get started

Article Tags