Viya Jobs that Create Multiple Output Files
If you have ever implemented a DDC that executes Viya Jobs, you certainly had to inspect the SAS log for clues about errors that might have happened, especially during the development phase, and even though you can ask the Viya Job to send back the SAS log, this most likely meant you had to modify portion of the JavaScript code that calls the job to set the parameter _debug=log, and in some cases the portion of the code that interprets the results received from the job as well.
When I first published the series of articles about integration of VA (SAS Visual Analytics) and Viya Jobs using DDC (Data-Driven Content object in VA), the third publication and the two subsequent use case examples used Viya Jobs that didn’t produce any visible output, but instead, they created tables that were loaded in CAS (memory). Because there was no visible output, the job ended up returning a small JSON structure to the DDC to inform the status of the SAS code execution, so after every SAS statement that could generate an error, we would call the macro function %stopOnError. This macro function does exactly what is says: it checks for errors in a few macro variables and if the error is found, it sends back to the DDC a small JSON message and aborts the job execution:
Even though it works, there are some limitations on this approach:
- You have to modify the SAS code to call the macro function after every step that may throw an error.
- The error message, without a broader context or the exact location where the error occurred, may not be sufficient, which would still require to modify the code to call the job with _debug=log.
- It only worked for Viya Jobs that didn’t send any content back to the DDC, such as SAS ODS (Output Delivery System) output, files, etc. If any other content had to be returned from the job to the DDC, then we could not send the JSON message back.
Well, it turns out that a Viya Job can return a list of output files in a JSON structure, and by leveraging that capability, we can present a new template of DDC + Viya Job that is a lot more flexible to return any content that you need, including the complete SAS log, while keeping the SAS code simple.
The advantages are:
- You don’t need to modify the SAS code by adding macro function calls in strategic locations to capture eventual errors.
- Any error in the SAS code is readily available in the SAS log file that can be retrieved and presented.
- It works with any kind of Viya Job, independently of the number and type of output it generates.
- You ultimately decide what to do with the output files: display in the web client (such as VA), download to your local computer, display in the browser’s console, etc.
Some disadvantages are:
- The output files generated with the SAS code must be stored in a location where it can be retrieved, and that normally includes content generated with ODS output. That means you will need extra ODS statements in the SAS code to store the ODS output in files.
- You can no longer send content to the DDC, such as ODS output that the browser understands, and hope the browser will automatically display it. You now control what to do with the different output files, which could also be an advantage (see #4 above) – it depends on how you look at it. You need to use JavaScript to post-process the JSON data and appropriately handle the list of output files, to give them their final destination.
Because a DDC is implemented as an HTML file, and a Viya Job can have a form (HTML file) associated with it, whenever there is a DDC that calls a job, I like to deploy the DDC as the job form and keep them together. The key difference between a DDC that calls a job and a job form, in terms of functionality, is that the DDC communicates with VA by sending and receiving messages with data, parameters, selections, etc. Everything else is the same. A DDC could for example receive data from VA and upload that data as a table to be processed by the Viya Job, and the job could load an output table in CAS, so other VA objects could visualize its content, like seen in this series of articles. To keep the focus on the key points of this example, which is creating and handling multiple output files that the Viya Job creates, we will not be using a DDC. Instead, we will be using a simple job form with a text box to capture the name of the input table and a button to submit the job. Again, the job form could easily be transformed in a DDC and integrate with VA.
The example explored here produces HTML and PDF output. The HTML is to be displayed and the PDF is to be downloaded, just to demonstrate that you are now in control of what to do with what the job produces as output:
- Viya Job
- Checks for expected parameter (input table in the format library.table) and returns an application error if something is wrong.
- Produces the HTML and PDF output files (proc print of 10 first observations).
- Job form (remember that it could be a DDC)
- Acquires the input parameter (table name as library.table) and calls the Viya Job.
- Checks the job output for errors.
- If error, displays log information in the browser’s developer console
- If ok, processes the job output files (displays HTML and downloads PDF)
The Viya Job Code
All files generated by the job to be returned to the job form (DDC) must be saved in the Viya Content Server via a special filename statement:
The name is important because this is how the job form (DDC) identifies this file in the JSON structure that it receives from the job, as you will see later.
Because the job can send back multiple files, I have adapted the old macro function %stopOnError (it’s ok if this is the first time you are hearing about this macro function) to send messages at the application level in a file named “_appout.json”. The macro was renamed to %sendApplicationMsg and is now used to inform errors that the application can identify, and abort the job execution if something is wrong. For example, we can use this macro in case an expected parameter was not passed to the job, or if a table doesn’t exist, etc. At the end, if the job runs successfully, you can also use that same macro function to inform the job form (DDC) that the job has reached the end and finished with success. Differently from the macro %stopOnError that had to be called multiple times in the SAS code, %sendApplicationMsg is only called if needed, according to the application logic. More precisely, in our example this macro function is used whenever a job input parameter is missing or the input table doesn’t exist. If you are wondering, we have a better mechanism to get information about unforeseen errors that will be explained later. This is an example of application message written to “_appout.json”:
The other two files that the job creates are named “_webout.html” and “_webout.pdf”, containing respectively the HTML and PDF output content. Those output formats are obtained via ODS. If you inspect the code, you will see that all ODS outputs are closed at the beginning of the code, and just before the proc print that generates a table with the first 10 observations is executed, two ODS outputs are open, one for HTML5 and one for PDF.
How those files are sent back to the job form (DDC) in a JSON structure depends on how the job is called, and it will be discussed in the next two sections: Calling the Job and Processing the Job Output.
This is the entire SAS code for our example:
The Job Form (DDC) Code
Calling the Job
The URL of the job form (DDC) is the URL of the Viya Job, with the parameter _action=form, which we normally add in the job properties, so we don’t have to pass that parameter all the time, keeping the job URL simpler. When the job form (DDC) calls the job, we would normally overwrite _action=form by explicitly passing _action=execute. But when we want the job to return a list of output files in JSON format, we must call the job with _action=json. Making a call to the job with _action set to json is the first key change.
This is an example of JSON with output files that you get when the job is called with _action=json:
Except for the log file, which name is automatically created for you, the other names are defined by you, and they are set in the SAS code with the filename statement.
There are two other parameters passed to the job that we need to highlight:
- _resultfile: By default, only file names matching _webout.* are included in the JSON output. If you want anything else to be included, you need to inform in this parameter. Wildcards “*” and “?” are accepted and multiple values can be separated by “,” (see documentation for more details). if _resultfile is such that it allows the SAS log to be included, a file named <SYS_COMPUTE_JOB_ID>.log is automatically added to the output, where <SYS_COMPUTE_JOB_ID> is the job ID and matches a macro variable with the same name in the job code. The *.log file is in JSON format and it contains more than just the SAS log lines, therefore a bit of post processing is needed. The extra information can be used to color code the log, in case you want to display the log in HTML format for example.
- _omittextlog: The default value of _omittextlog is true, but if it’s set to false, an additional SAS log file named <SYS_COMPUTE_JOB_ID>.log.txt is also automatically included in the output, as long as _resultfile permits. The *.log.txt file only contains the SAS log lines in text format and it’s easier to consume. We are not generating this file in our example.
Lastly, the header of the HTTP request must have “Accept” set to “application/json”, because this is what the Viya Job returns with _action=json.
In our example, all those steps are performed in the callJob() function (highlighted in yellow).
Processing the Job Output
Once the job finishes and returns execution to the job form (DDC), you need to inspect its response as a JSON. But what if there was an error when the job was executing that you were not able to anticipate and inform it through a call to the macro function %sendApplicationMsg? Well, calling the job with _action=json has another advantage: if there is an error in the SAS code execution, a special JSON output is automatically sent back, according to the documentation:
As you can see, the attributes in this JSON can be used to not only detect there was an error, for example, by inspecting the “errorCode”, but also to have additional insights on the error, including the entire SAS log. So once the job returns execution to the job form (DDC) and an error is detected, you can dump the SAS log in the browser’s developer console for easy troubleshooting (please see code highlighted in purple above). Obviously, you can skip this step after development is concluded and the solution is deployed in production.
Note: in Viya3.5, the response received when there is an error executing the SAS code is slightly different and is documented here. It contains not only the error message, but also the JSON structure with the result files that were specified in the _resultfile parameter. This is important because instead of reading the response as a JSON, you need to read it as a text and then parse its components into their own JSON objects.
If there was no error, the JSON structure received is actually the job output, with URI to the output files in it, as specified in the _resultfile parameter, so the next step is to inspect the content of the file we called _appout.json to see if there was no error at the application level. Remember that application errors are those that you generated when validating input parameters and are different from unexpected execution errors. If there was no application error, then you should access the output files URI, extract their content, and give them the proper destination: display the HTML and download the PDF. In our example, these are all performed in the Submit() function:
To download the PDF file, you call the function downloadFile(). This function uses JavaScript to create a <a> element on the fly and simulate a user clicking in its hyperlink, which points to the PDF file’s URI.
Deploying and Running the Example
The complete SAS code and job form are attached in a zip file. Once you add them to a Viya Job, you can run it directly from SAS Studio or SAS Job Execution web applications. Remember to set the parameter _action=form in the Viya Job properties.
The following two videos show how to create and run the job with the provided files in SAS Job Execution web application, and how to easily check the SAS log in the browser’s developer console (Ctrl+Shift+i) when there is an error in the SAS code.
