In one of the many customer iterations I had recently, I was asked to help develop a solution to allow an image that was dynamically created by a third-party library/API to be embedded in a SAS Visual Analytics (VA) report. In addition to some input parameters passed to that external API that are irrelevant to our discussion, there was one in special that contained the full path and name of the output image to be generated. Although it seems a simple task, it doesn’t take too long to realize some of the challenges:   If the output image is stored with a fixed path and name, it doesn’t work well in VA reports with concurrent users: Image file names and/or locations must be unique and dynamically generated to allow concurrency, so different users don’t overwrite each other. If you generate random distinct names for each request, after the solution has been running for a while, you may have many image files stored somewhere: You must have a mechanism that manages the number of images stored, so it doesn’t grow forever. The client application consuming the images runs on a browser, so we must be able to stream the images or reference them by a URL or URI: The third-party API doesn’t accept URL or URI as parameters, so you must perform a conversion of some sort.   I’ve decided to use SAS code as a Viya Job to be able to deal with the constraints above. With the SAS code I’d be able to:   Leverage temporary files via filename statements to generate the image names – those file names are unique, and files get automatically deleted when the SAS session ends, helping with challenges 1 and 2. Leverage Viya Content Server to temporarily store the image files – files stored in the Content Server can be referenced by a URI and are automatically deleted after some time, helping with challenges 2 and 3.   I only needed to find a way to transfer the image generated by the external API to the Viya Content Server, and it only took me one Google search for “sas copy binary files” to find the solution I was looking for in the first hit, a blog by Chris Hemedinger, more precisely the portion that uses the FCOPY function from SAS:   https://blogs.sas.com/content/sasdummy/2013/09/17/copy-file-macro/   This is the SAS code with the key elements: * Close all ODS output; ods _all_ close; * Set the temporary file location; filename _bcin TEMP recfm=n; * RECFM=N means that the input is read as a stream (there are no records); * Obtain the temporary /path/file value; %let tmpfile=%qsysfunc(pathname(_bcin)); * Here you would add the code to call the external API passing &tmpfile as parameter; * Fileref _webout with name _webout.html is automatically assigned in Viya Jobs; * If fileref with name _webout.* has an extension that the browser knows how to visualize, such as html, jpg, etc. it will display it; * The pre-assigned fileref _webout is not needed, so we must delete it first; %let RC=%sysfunc(fdelete(_webout)); * Set content server file location; * We need a fileref with the image's extension (so the browser can display it) and add recfm=n; * The fileref could be called anything you want; filename _webout filesrvc parenturi="&SYS_JES_JOB_URI" name="_webout.jpg" recfm=n; * Copy API output to Content Server; data _null_; length msg $ 384; rc=fcopy('_bcin', '_webout'); if rc=0 then put 'NOTE: Copied _bcin to _webout.'; else do; msg=sysmsg(); put 'ERROR:' rc= msg=; end; run; The code starts with closing all ODS outputs because we know we will be returning an image produced by an external API, so no ODS output is needed. Then we set the temporary file name and location using a TEMP fileref called _bcin and store its value in a macro variable. That macro variable is passed to the external API as the full path and name of the output image to be created. Next, we delete the existing fileref _webout. This fileref is automatically created for you in Viya Jobs with attribute name=_webout.html. All content of filerefs with name=_webout.* are automatically streamed back to the calling application, but we need a fileref that is slightly different. We need a fileref with RECFM=N, to indicate the file consists of a stream of bytes in binary format with no record boundaries, which is going to be necessary later when using the function FCOPY to copy the image generated by the API to the Content Server, and we want its name=_webout.jpg, so the browser knows it’s an image and therefore knows how to display it. The fileref itself does not need to be called _webout. The final data step copies the output image to Viya Content Server leveraging both filerefs. For the cleanup, the temporary file given by the TEMP fileref will be automatically disposed when the job finishes, and its copy in the _webout fileref will also be automatically deleted after some time.   If for some reason you want the name attribute of the fileref to be something other than _webout.*, or send back multiple files to the calling application, then you can call the Viya Job with _action=json. This makes the job return a list of output files in a json structure and the calling application has the freedom to decide what to do with each output file. This is explored in this other SAS Communities publication called Viya Jobs that Create Multiple Output Files.   In my final implementation, the code returned a couple of files (the output image and a small json file with information about the code execution status), simply because this is the template I’m currently leveraging for the Viya Jobs I develop. It also makes it simple to integrate the job with VA via DDC (Data-Driven Content), which is what I had to do for that customer. Indeed, I used that framework to provide a solution where the chart was being created in Python, using SAS PROC PYTHON, after receiving input parameters from VA, but this will be the topic for another blog. ... View more

In the series of articles that talked about integrating VA (SAS Visual Analytics) with SAS Viya Jobs via DDC (Data-Driven Content) objects, you saw a few examples where Viya Jobs were used to create CAS tables on the fly to be consumed by VA reports. To have the VA report refreshed with the new table content generated by the job, you leveraged VA’s capability to refresh objects. Figure 1- Typical flow using auto-refresh feature   At a high level, the process can be described as (please, refer to the previous animated gif): The DDC object in VA has an URL that points to the SAS Viya Job, which is configured with _action=form, so the Viya Job Form takes over the Viya Job Code execution and the form becomes the DDC implementation. Also, the other VA object that uses the CAS table created by the SAS Viya Job is configured with auto-refresh, which means that the object makes recurrent checks for table updates. A change or action in the report that leads to a change in the data that gets passed to the DDC triggers the DDC execution: a JSON message with data and/or parameters is passed from VA to the DDC implementation code (JavaScript). The DDC implementation code processes the JSON message and eventually makes an HTTP call to execute the Viya Job Code with _action=execute (or _action=json). The Job executes and updates the CAS table. Next time the VA object configured with auto-refresh checks for changes on the CAS table it identifies the table has changed and fetches its content to refresh the object. Although this works perfectly, there are a few things you need to be aware of: The refresh feature only works for reports open in view mode – it’s not available when you are editing the report. As a report author, you would need to go back and forth between the edit and view mode to test. This has been mentioned in the series of articles presented in the beginning. You cannot prevent the auto-refresh from happening at the exact moment that the table is being recreated by the job code. If VA tries to fetch the table to refresh the object at the exact same time that the table has been dropped to be updated, the VA report issues an error. Because this is a rare event, this has not been mentioned in the series of articles for simplicity, but you should always account for this possibility to prevent occasional errors. One way you can prevent this from happening is by guaranteeing that the table always exists, and you can do that by working on a temporary table, and when the table is ready to be updated, you delete the existing records (instead of dropping the table) and append the temporary table into the empty table. This is not a big deal, but it adds extra steps to the job code. VA objects that are configured for automatic refresh are not the best candidates for user interactivity simply because the user never knows the exact moment the object will be refreshed. This article presents a solution that lets you refresh VA report objects on demand, and it solves all three limitations above. I must highlight, though, that those limitations are not critical at all, and you will be just fine if you leverage the out-of-the-box auto-refresh feature that VA provides, as long as you can guarantee that inexistent tables aren’t fetched automatically by VA report objects.   For those who want to take a different spin on how a DDC implementation can be used to control when VA report objects should fetch their CAS table data and get their content refreshed, keep reading.   The principle behind the solution is simple and relies on selection messages sent from the DDC to the VA report. If you want to know more about the basics of DDC and the messages exchanged with VA, please refer to this publication.   At a high level, the solution works like this: whenever the job ends execution successfully (meaning new data is available in the CAS table created by the job), the DDC sends a message back to VA informing that certain rows are being selected, and all report objects that are setup as target of a filter action, where the DDC is the source, will use the selected item from the message as the filter criteria to fetch data from the CAS table, and that refreshes the object.   Figure 2- Typical flow using selection message   The previous animated gif illustrates the process, and as you can see, the difference, compared to the auto-refresh approach discussed earlier, relies on the filter action from the DDC to the object to be refreshed, and the selection message sent at the end (step #4).   The selection message tells VA that that certain data has been selected in the DDC. To help you understand, imagine the DDC implementation uses the data assigned to its Roles pane to produce a bar chart. That bar chart shows invoice by region (Asia, Europe, Africa, etc.). If you wanted to allow users to click in the bar chart and let VA know for example that Asia had been selected, you would have to set an event handler in the DDC bar chart implementation to detect when a bar has been clicked on, and send a selection message to VA informing which region had been selected so that other VA report objects that have actions associated with the DDC could act accordingly (such as filter the object to display only data for Asia). A selection message is a simple JSON structure that contains two values: the ID of the DDC object (so VA knows the origin of the selection message and therefore knows the VA objects that have actions defined with that origin), and the data selected. The data selected is simply a list of rows, in reference to the data that the DDC received from VA: Figure 3- Example of selection message   Independently of what the DDC produces as the output (visible content like the previous bar chart or a table loaded in memory), we can always send selection messages to VA as if a chart had been clicked on.   You may be thinking that if the selection/filter of the data is what causes the VA object to refresh, how can we accomplish the refresh but display all the data, and not just a subset due to the filter being applied? That’s where the trick comes in.   Ideally the selection message would be such that it would tell the VA object that nothing had been selected (empty list of rows in the JSON message), and it would fetch all the data, but selecting nothing, which is the same as displaying all the data, is the default state of an object, and an object that is already displaying all the data will not be affected by a message that tells it to display all the data. The solution is to send two messages, separated by a time interval: the first message selects/filters a subset of the rows, the first row for example, and the second message selects/filters nothing, forcing all data to be fetched. It’s worth mentioning that the data/table that the DDC receives in the VA message and the CAS table that is created by the Viya Job and feeds other VA objects need to have a mapping, in other words, a data item that is common to those two tables, so the filter can occur (we will be talking more about it in a moment). Figure 4- Two selection messages to implement the refresh   Let’s take a closer look into the JavaScript code that sends those two messages, forcing the VA object to refresh. Figure 5- JavaScript fuction ForceVAObjectRefresh   The function forceVAObjectRefresh() implements what we have just described. It receives two arguments: resultName and millisecondsBeforeRefresh. The argument resultName is the ID of the DDC, which came in the JSON message that the DDC received from VA, and millisecondsBeforeRefresh is the amount of time you want to wait between the two selection messages that the DDC sends to VA. It was set as an argument to the function because you may need to tune its value according to your environment. You want to give it enough time for the first selection message to be processed by VA, before the second message is sent, otherwise the filter action that the DDC has with other objects may not be triggered. Usually, 1 or 2 seconds should work. To send the messages, you leverage the function va.messagingUtil.postSelectionMessage(), provided to you in messagingUtil utility file available in GitHub. The first time the function is called, you select row 0 (the first row). The second time the function is called you select nothing, which is interpreted by VA as no filter applied, but this second call only happens after the time you specified in parameter millisecondsBeforeRefresh, which is passed internally to function setTimeout(). Due to the asynchronous nature of this process, a Promise is being used to allow you to properly act once the function is fully executed, such as write the message “Done!” in the VA report if you want.   NOTE: function forceVAObjectRefresh() is available in GitHub in the messagingUtil.js utility file.   So far so good, but there is one more thing to discuss: the data items assigned to the DDC Roles pane, that in turn define the columns of a summary table in the JSON message that the DDC receives from VA, and the CAS table generated/updated by the Viya job not necessarily have the same columns, so how does the filter occur, if there may be no mapping between those tables? This is what you will see next.   All DDC objects must have at least one data item assigned in their Roles pane. Like said, the items assigned in the DDC Roles pane determine the columns passed from VA to the DDC, and the data is aggregated by the categorical columns (like a group by in SQL). If no categorical column exists, the DDC receives a table with only one row (the aggregated values for each numeric column that have been assigned). For this solution to work, you must have at least one categorical data item assigned to the DDC. The reason for that is because you need a data item that can be used to filter the output CAS table that feeds the VA objects, and only categorical data items are used in filter actions. When you are already sending data to your DDC, by assigning them to the DDC Roles pane, this requirement of at least one categorical data item may be already met, and when the DDC sends a message back to VA to filter the first row, it is the first row of the table that VA had sent to the DDC, so VA knows the values of the categorical data items of that first row. If the data item assigned to the DDC Roles pane has at least one similar column compared with the table that the Viya Job creates and stores in CAS, the action filter is already functional, but if the columns on that table have no corresponding data items in the DDC roles assignment, you must define their relationship via table mapping.   Independently if the mapping already exists or if you need to explicitly define it, the value of the categorical data item in the first row may not exist in the CAS table created by the Viya Job. When this happens, you see a brief warning message “No data matches the current filter” right after the first message is processed by VA and until the second selection message finishes the refreshing by bringing all the data, which is not ideal. Figure 6- Filtering first row may produce an empty result set   In addition, it turns out there might not even be a mapping between those tables. What do you do in that case? Fortunately, the solution is simple: create a categorical calculated item that contains a fixed value (all values the same for all rows, such as “anything”) and assign this calculated item in the DDC Roles pane, then create the same categorical calculated item in the table that came from the Viya Job. This column becomes the natural table mapping column. In fact, creating that VA report calculated item in both tables (the table with data items assigned to the DDC Roles pane and the CAS table generated by the Viya Job), passing that calculated item to the DDC, and using it in the table mapping is highly recommended, even if a good candidate for a table mapping column already exists. This is because it contributes for a smoother data refresh in VA, either because it guarantees there will always be data to be filtered (the result set is never empty therefore no warning message is ever displayed), or because it produces the same result set, which includes all the data, whenever the first or second message that the DDC sends to VA is processed: remember that all values are the same in that calculated item, so if you filter the CAS table by the value of the calculated item in the first row of the aggregated table passed to the DDC, or if you don’t filter the CAS table at all, you will always get all the records in the result set, and the refresh experience is much better because there is no change when the second selection message is processed, although it is still needed for the solution to work. Besides, you have the chance to name the calculated item appropriately, such as “Refresh”, or even something more explainable like “For data refreshing”, to remind you that that data item is there solely for implementing the data refresh. Figure 7- Filtering first row is the same as selecting all rows   To summarize, these are the things you need to do (supposing you ran the Viya Job at least once and the Viya Job output CAS table already exists and was already added to the VA report): Create a categorical calculated item in the VA report with a fixed value (e.g. “anything”) in both the table that feeds the DDC as well as the one that was produced by the Viya job and feeds the VA objects to be refreshed. Make sure the calculated item is used in the table mapping between those two tables. Assign the calculated item to the DDC Roles pane. In the DDC object, define a filter action with the VA objects to be refreshed. Modify your DDC implementation code to call the function forceVAObjectRefresh() when the Viya Job finishes execution. One more thing… when users open the VA report for the very first time, it might happen that the Viya Job output table doesn’t exist yet. In that case, the VA object will not be refreshed once the job finishes because initially there was no data in the object to be filtered. Once the job is executed and the table is created, users should just reopen the report, and they are back in business.   For an example of VA report and complete DDC implementation as a Viya Job, please check the attached example.zip file. Just start with the readme.txt file and follow the steps. The deployment of this example could have been simplified if we used a package, but it would require admin rights to import the package, so I’ve opted to provide you with the individual files. It’s a little bit more work to deploy it, but it’s good to expose the components that make up this solution. This example should work for Viya 3.5 and beyond. ... View more

A customer once asked me how to use a text box to filter multiple values. He wanted to type a list of zip codes and use that list to filter other objects in the report. If he had Viya 2020.1.1 (December 2020) or later, I could have referred him to the search feature for List control objects, but he was running Viya 3.5, so we had to be creative and develop our own solution.   The solution was obviously an advanced filter applied to the object to be filtered, and the Contains operator was the fist thing that came up to my mind. In Viya 3.5, the syntax for Contains is the following:   'Parameter'p Contains 'ZipCode'n   This of course was the solution at a high level, but it didn’t take too long to realize we would need to:   Define a separator for the values in the list. Take care of certain exceptions, such as when the Text Input control is empty. Deal with blank spaces that could appear at the beginning and/or at the end of typed zip code values. Deal with mixed upper and lower case characters to support all types of zip codes around the world that could contain letters. Refine our requirements – for example: what happens if an incomplete zip code is typed, such as 234? Should we select all zip codes that contain 234 or should it be a full match? What about the opposite – user types 12345-678 and the table contains 12345 – should they match?   Here is what we have decided:   Using comma as the separator would work just fine, as comma is not a character used in zip codes. If the Text Input control is left empty, then it should have the same behavior as Visual Analytics: no filter is applied. Leading and trailing blanks for each of the values should be ignored, as they are not part of the zip code values being searched. We would standardize in upper case characters only. Typed values and values stored in the table should match entirely, so 12345 would not match 12345-678 and vice-versa.   To test if the input list of zip codes (a character parameter) is empty, we used the IsSet operator, and to ignore the filter if it’s empty, we used the expression 1 = 1 (which returns TRUE, as Visual Analytics doesn’t have a Boolean type):   IF ( 'Parameter'p IsSet ) RETURN (<expression>) ELSE ( 1 = 1 )   Now we needed to work on the <expression>. To remove blanks, we used the RemoveBlanks function. We could have used the RemoveChars function instead, which is more generic and would work for other types of characters. To standardize the input, we applied the UpCase function:   UpCase(RemoveBlanks('Parameter'p, _All_))    Our <expression> then became:   UpCase(RemoveBlanks('Parameter'p, _All_)) Contains UpCase(RemoveBlanks('ZipCode'n, _All_))   Observe that the same transformations were applied to both sides, just to make sure we are comparing apples to apples.   For the full match of individual values, the operator Contains naturally takes care of cases where the user types 12345 on the list of values and the table contains a zip code like 12345-678 (it’s a no match as defined), but the opposite is not true:   “zipcodeA,12345,zipcodeB” Contains “12345-678” --> returns FALSE (not a match): correct “zipcodeA,12345-678,zipcodeB” Contains “12345” --> returns TRUE (a match): incorrect   If the user types 12345-678 in the Text Input control, it would match zip code 12345, and potentially any other subset of the input value, such as those bad zip codes:   12345- 123 45-6 -678 - 7 etc…   To guarantee the full match of what was being typed, we had to search for the entire value, and an easy way to do that was to include characters to mark the beginning and the end of the values in the search string. Well, it turned out that by definition, all values were already separated by comma in the input text, so it naturally became the character to indicate start and end of values. All we had to do was to include the separator in the expression, by concatenating a comma before and after the zip codes stored in the table, using the operator Concatenate twice:   Concatenate( ',', Concatenate( UpCase(RemoveBlanks('ZipCode'n, _All_)), ',' ) )   After concatenating comma before and after:   “zipcodeA,12345, zipcodeB” Contains “,12345-678,” --> returns FALSE (not a match): correct “zipcodeA,12345-678, zipcodeB” Contains “,12345,” --> returns FALSE (not a match): correct   This seemed all we had to do, except for one thing: if user types 12345-678 as the first or the last element on the search list, or if it’s the only element in the list, it will not contain the separator character, and valid zip code values will not be found:   “12345-678,zipcodeB” Contains “,12345-678,” --> returns FALSE (not a match): incorrect “zipcodeA,12345-678” Contains “,12345-678,” --> returns FALSE (not a match): incorrect “12345-678” Contains “,12345-678,” --> returns FALSE (not a match): incorrect   To fix that we needed to append a comma at the beginning and at the end of the entire typed input parameter string, exactly as we did before with the zip codes stored in the table:   Concatenate( ',', Concatenate( UpCase(RemoveBlanks('Parameter'p, _All_)), ',' ) )   After we did that:   “,12345-678,zipcodeB,” Contains “,12345-678,” --> returns TRUE (a match): correct “,zipcodeA,12345-678,” Contains “,12345-678,” --> returns TRUE (a match): correct “,12345-678,” Contains “,12345-678,” --> returns TRUE (a match): correct   Putting everything together, this is how the filter expression looked like (before Viya 2023.06 syntax):   IF ( 'Parameter'p IsSet ) RETURN ( Concatenate(',', Concatenate(UpCase(RemoveBlanks('Parameter'p, _All_)), ',')) Contains Concatenate(',', Concatenate(UpCase(RemoveBlanks('ZipCode'n, _All_)), ',')) ) ELSE ( 1 = 1 )    Same expression starting with Viya 2023.06 syntax (basically IsSet and Contains are functions instead of operators and the key work _All_ no longer has underscores):   IF IsSet('Parameter'p) RETURN Contains( Concatenate(',', Concatenate(UpCase(RemoveBlanks('Parameter'p, All)), ',')), Concatenate(',', Concatenate(UpCase(RemoveBlanks('ZipCode'n, All)), ',')) ) ELSE ( 1 = 1 )    In our case, zip codes were already stored as characters in the source table, but if that was not the case, we could simply apply the Format function to transform them into characters, and the rest of the expression would be the same.   Obviously, as long as the requirements remain the same, this expression can be leveraged for other use cases, such as searching for car makes.   Suppose that you have a bar chart, and you want to be able to type one or more vehicle Makes separated by comma in the Text Input control object at the top to filter the chart: Figure 01-Text input and bar chart objects Let’s take a look at how you would do that in three easy steps.   FIRST, you create a parameter with the following attributes:   Name: list of Makes separated by comma Type: Character Multiple values: unchecked Current value: empty Figure 02-Create character parameter Knowing that this parameter will carry multiple values, you may be attempted to check the box for multiple values, but a text input control object, where this parameter is going to be assigned to receive its value, does not accept multiple values – the object is a text box, and therefore it accepts only a single value, which is a string.   SECOND, you assign the parameter to the Text Input control object. Note that before Viya 2024.05, the parameter is assigned in the object’s Roles pane: Figure 03-Assign parameter to text input object prior to Viya 2024.05    Starting in Viya 2024.05, this assignment is done in the object’s Actions pane:  Figure 04-Assign parameter to text input object in Viya 2024.05 and beyond   THIRD, you add the advanced filter discussed previously into the bar chart’s Filters pane – remember that the name of the parameter and the data item being searched are different in this example.   Now let’s suppose you type “Toyota,,   ,    bmw,  hoNdA  ,Mercedes,Volks Wagen     , xyz, Nissan SUV, Land rover”, without the quotation marks, in the Text Input object and hit Enter – you must hit Enter for the value to be assigned to the parameter. Starting from the inner most operators/functions and moving outwards, this is what happens:   Remove all blanks (space characters) from the input parameter. You should get this: Toyota,,,bmw,hoNdA,Mercedes,VolksWagen,xyz,NissanSUV,Landrover Make everything upper case as a best practice for string comparison: TOYOTA,,,BMW,HONDA,MERCEDES,VOLKSWAGEN,XYZ,NISSANSUV,LANDROVER Concatenate a comma at the end. Note that comma is used as the separator. It must be a character or combination of characters that do not appear in the data being filtered - all possible values that Make can assume in this example. This is what you have so far: TOYOTA,,,BMW,HONDA,MERCEDES,VOLKSWAGEN,XYZ,NISSANSUV,LANDROVER, Concatenate a comma (or the other elected separator) at the beginning. Let’s call this result the standardized parameter: ,TOYOTA,,,BMW,HONDA,MERCEDES,VOLKSWAGEN,XYZ,NISSANSUV,LANDROVER, Repeat the same steps for the Make column to create the standardized Make, so you can compare apples to apples. For example, if the value of the Make being compared is Land Rover, it would become: ,LANDROVER, Check if the standardized parameter contains the searched standardized Make. Only do all of that if the parameter is set. If it’s not set, return true, meaning all Make values should be displayed, which is the default in SAS Visual Analytics. Figure 05-Results  Note that Mercedes did not show up in the bar chart because it’s stored as Mercedes-Benz in the table. In another hand, Nissan is a valid car make but Nissan SUV is not, so it was not returned. Also, because the implemented filter is not case sensitive, Honda was displayed as expected, even if typed with mixed casing in a strange way. Because spaces are eliminated, mistakenly typed Volks Wagen (two words), matched the make Volkswagen. Non-existing makes such as xyz, empty values, and extra blank spaces didn’t affect the results.   Please, let me know in the comments what you think about this solution and how or for what purpose you have used it. If you have Viya release 2020.1.1 (December 2020) or above, I’m particularly interested in knowing why you had to adopt this solution instead the List control object with search capabilities available out of the box.         ... View more

If you have ever implemented a DDC that executes Viya Jobs, you certainly had to inspect the SAS log for clues about errors that might have happened, especially during the development phase, and even though you can ask the Viya Job to send back the SAS log, this most likely meant you had to modify portion of the JavaScript code that calls the job to set the parameter _debug=log, and in some cases the portion of the code that interprets the results received from the job as well.   When I first published the series of articles about integration of VA (SAS Visual Analytics) and Viya Jobs using DDC (Data-Driven Content object in VA), the third publication and the two subsequent use case examples used Viya Jobs that didn’t produce any visible output, but instead, they created tables that were loaded in CAS (memory). Because there was no visible output, the job ended up returning a small JSON structure to the DDC to inform the status of the SAS code execution, so after every SAS statement that could generate an error, we would call the macro function %stopOnError. This macro function does exactly what is says: it checks for errors in a few macro variables and if the error is found, it sends back to the DDC a small JSON message and aborts the job execution:   Figure 01-Example of old error message   Even though it works, there are some limitations on this approach: You have to modify the SAS code to call the macro function after every step that may throw an error. The error message, without a broader context or the exact location where the error occurred, may not be sufficient, which would still require to modify the code to call the job with _debug=log. It only worked for Viya Jobs that didn’t send any content back to the DDC, such as SAS ODS (Output Delivery System) output, files, etc. If any other content had to be returned from the job to the DDC, then we could not send the JSON message back. Well, it turns out that a Viya Job can return a list of output files in a JSON structure, and by leveraging that capability, we can present a new template of DDC + Viya Job that is a lot more flexible to return any content that you need, including the complete SAS log, while keeping the SAS code simple.   The advantages are: You don’t need to modify the SAS code by adding macro function calls in strategic locations to capture eventual errors. Any error in the SAS code is readily available in the SAS log file that can be retrieved and presented. It works with any kind of Viya Job, independently of the number and type of output it generates. You ultimately decide what to do with the output files: display in the web client (such as VA), download to your local computer, display in the browser’s console, etc.   Some disadvantages are: The output files generated with the SAS code must be stored in a location where it can be retrieved, and that normally includes content generated with ODS output. That means you will need extra ODS statements in the SAS code to store the ODS output in files. You can no longer send content to the DDC, such as ODS output that the browser understands, and hope the browser will automatically display it. You now control what to do with the different output files, which could also be an advantage (see #4 above) – it depends on how you look at it. You need to use JavaScript to post-process the JSON data and appropriately handle the list of output files, to give them their final destination. Because a DDC is implemented as an HTML file, and a Viya Job can have a form (HTML file) associated with it, whenever there is a DDC that calls a job, I like to deploy the DDC as the job form and keep them together. The key difference between a DDC that calls a job and a job form, in terms of functionality, is that the DDC communicates with VA by sending and receiving messages with data, parameters, selections, etc. Everything else is the same. A DDC could for example receive data from VA and upload that data as a table to be processed by the Viya Job, and the job could load an output table in CAS, so other VA objects could visualize its content, like seen in this series of articles. To keep the focus on the key points of this example, which is creating and handling multiple output files that the Viya Job creates, we will not be using a DDC. Instead, we will be using a simple job form with a text box to capture the name of the input table and a button to submit the job. Again, the job form could easily be transformed in a DDC and integrate with VA.   The example explored here produces HTML and PDF output. The HTML is to be displayed and the PDF is to be downloaded, just to demonstrate that you are now in control of what to do with what the job produces as output: Viya Job Checks for expected parameter (input table in the format library.table) and returns an application error if something is wrong. Produces the HTML and PDF output files (proc print of 10 first observations). Job form (remember that it could be a DDC) Acquires the input parameter (table name as library.table) and calls the Viya Job. Checks the job output for errors.       If error, displays log information in the browser’s developer console       If ok, processes the job output files (displays HTML and downloads PDF)   The Viya Job Code   All files generated by the job to be returned to the job form (DDC) must be saved in the Viya Content Server via a special filename statement: Figure 02-Filename statement     The name is important because this is how the job form (DDC) identifies this file in the JSON structure that it receives from the job, as you will see later.   Because the job can send back multiple files, I have adapted the old macro function %stopOnError (it’s ok if this is the first time you are hearing about this macro function) to send messages at the application level in a file named “_appout.json”. The macro was renamed to %sendApplicationMsg and is now used to inform errors that the application can identify, and abort the job execution if something is wrong. For example, we can use this macro in case an expected parameter was not passed to the job, or if a table doesn’t exist, etc. At the end, if the job runs successfully, you can also use that same macro function to inform the job form (DDC) that the job has reached the end and finished with success. Differently from the macro %stopOnError that had to be called multiple times in the SAS code, %sendApplicationMsg is only called if needed, according to the application logic. More precisely, in our example this macro function is used whenever a job input parameter is missing or the input table doesn’t exist. If you are wondering, we have a better mechanism to get information about unforeseen errors that will be explained later. This is an example of application message written to “_appout.json”:   Figure 03-Example of new application error message   The other two files that the job creates are named “_webout.html” and “_webout.pdf”, containing respectively the HTML and PDF output content. Those output formats are obtained via ODS. If you inspect the code, you will see that all ODS outputs are closed at the beginning of the code, and just before the proc print that generates a table with the first 10 observations is executed, two ODS outputs are open, one for HTML5 and one for PDF.   How those files are sent back to the job form (DDC) in a JSON structure depends on how the job is called, and it will be discussed in the next two sections: Calling the Job and Processing the Job Output.   This is the entire SAS code for our example:   Figure 04-Viya Job code   The Job Form (DDC) Code Calling the Job   The URL of the job form (DDC) is the URL of the Viya Job, with the parameter _action=form, which we normally add in the job properties, so we don’t have to pass that parameter all the time, keeping the job URL simpler. When the job form (DDC) calls the job, we would normally overwrite _action=form by explicitly passing _action=execute. But when we want the job to return a list of output files in JSON format, we must call the job with _action=json. Making a call to the job with _action set to json is the first key change.   Figure 05-Calling Viya Job with HTML form and _action=json   This is an example of JSON with output files that you get when the job is called with _action=json:   Figure 06-Example of JSON response with list of job output files   Except for the log file, which name is automatically created for you, the other names are defined by you, and they are set in the SAS code with the filename statement.   There are two other parameters passed to the job that we need to highlight:    _resultfile: By default, only file names matching _webout.* are included in the JSON output. If you want anything else to be included, you need to inform in this parameter. Wildcards “*” and “?” are accepted and multiple values can be separated by “,” (see documentation for more details). if _resultfile is such that it allows the SAS log to be included, a file named <SYS_COMPUTE_JOB_ID>.log is automatically added to the output, where <SYS_COMPUTE_JOB_ID> is the job ID and matches a macro variable with the same name in the job code. The *.log file is in JSON format and it contains more than just the SAS log lines, therefore a bit of post processing is needed. The extra information can be used to color code the log, in case you want to display the log in HTML format for example. _omittextlog: The default value of _omittextlog is true, but if it’s set to false, an additional SAS log file named <SYS_COMPUTE_JOB_ID>.log.txt is also automatically included in the output, as long as _resultfile permits. The *.log.txt file only contains the SAS log lines in text format and it’s easier to consume. We are not generating this file in our example.   Lastly, the header of the HTTP request must have “Accept” set to “application/json”, because this is what the Viya Job returns with _action=json.   In our example, all those steps are performed in the callJob() function (highlighted in yellow).   Figure 07-JavaScript function callJob()     Processing the Job Output   Once the job finishes and returns execution to the job form (DDC), you need to inspect its response as a JSON. But what if there was an error when the job was executing that you were not able to anticipate and inform it through a call to the macro function %sendApplicationMsg? Well, calling the job with _action=json has another advantage: if there is an error in the SAS code execution, a special JSON output is automatically sent back, according to the documentation:   Figure 08-Example of JSON response when there is an error in the SAS code   As you can see, the attributes in this JSON can be used to not only detect there was an error, for example, by inspecting the “errorCode”, but also to have additional insights on the error, including the entire SAS log. So once the job returns execution to the job form (DDC) and an error is detected, you can dump the SAS log in the browser’s developer console for easy troubleshooting (please see code highlighted in purple above). Obviously, you can skip this step after development is concluded and the solution is deployed in production.   Note: in Viya3.5, the response received when there is an error executing the SAS code is slightly different and is documented here. It contains not only the error message, but also the JSON structure with the result files that were specified in the _resultfile parameter. This is important because instead of reading the response as a JSON, you need to read it as a text and then parse its components into their own JSON objects.   If there was no error, the JSON structure received is actually the job output, with URI to the output files in it, as specified in the _resultfile parameter, so the next step is to inspect the content of the file we called _appout.json to see if there was no error at the application level. Remember that application errors are those that you generated when validating input parameters and are different from unexpected execution errors. If there was no application error, then you should access the output files URI, extract their content, and give them the proper destination: display the HTML and download the PDF. In our example, these are all performed in the Submit() function:   Figure 09-JavaScript function Submit()     To download the PDF file, you call the function downloadFile(). This function uses JavaScript to create a <a> element on the fly and simulate a user clicking in its hyperlink, which points to the PDF file’s URI.   Figure 10-JavaScript function downloadFile()     Deploying and Running the Example   The complete SAS code and job form are attached in a zip file. Once you add them to a Viya Job, you can run it directly from SAS Studio or SAS Job Execution web applications. Remember to set the parameter _action=form in the Viya Job properties.   The following two videos show how to create and run the job with the provided files in SAS Job Execution web application, and how to easily check the SAS log in the browser’s developer console (Ctrl+Shift+i) when there is an error in the SAS code.     ... View more