How to Limit the Number of Input Rows in a DataFlux Data Management Studio Data Job

[#1]

Sometimes you may want to limit the number of rows you are reading in your DataFlux Data Management Studio Data Job. This is especially useful when trying to design and test your data jobs for large input data sets.  

There is an advanced property called MAX_OUTPUT_ROWS that can be used to limit the number of rows read. This property is available on all the Data Input nodes in a Data Management Studio data job. To view the advanced properties for a node, select the node in the data flow and then select > Advanced Properties from the toolbar. It is a best practice to set the value of MAX_OUTPUT_ROWS property using a macro variable as shown below.

Figure 1: MAX_OUTPUT_ROWS Advanced Property

Using a macro variable for this setting means you can easily change the number of rows you are reading to increase or decrease the number of records you are working with in your data job. In the example below, the macro variable has been set to 10; therefore, only 10 rows will be read in from the data source.

 

Figure 2:  NUM_ROWS_TO_READ Macro Variable Setting

As expected the preview of the node below shows only the first 10 records for the data source. Figure 3:  MAX_OUTPUT_ROWS Data Job Preview

Setting the macro variable value to 0 means that all rows will be read in from the input data source. Typically, in your macro variable settings on your DataFlux Data Management Server you will have the MAX_OUTPUT_ROWS macro variable name you used in your data jobs set to 0, so all rows will be read in from you input data sources when running data jobs on the server.    

For more information on Advanced properties for nodes in DataFlux Data Management Studio, refer to the topic “Advanced Properties” in the DataFlux Data Management Studio 2.7: User’s Guide.