A customer once asked me how to use a text box to filter multiple values. He wanted to type a list of zip codes and use that list to filter other objects in the report. If he had Viya 2020.1.1 (December 2020) or later, I could have referred him to the search feature for List control objects, but he was running Viya 3.5, so we had to be creative and develop our own solution.   The solution was obviously an advanced filter applied to the object to be filtered, and the Contains operator was the fist thing that came up to my mind. In Viya 3.5, the syntax for Contains is the following:   'Parameter'p Contains 'ZipCode'n   This of course was the solution at a high level, but it didn’t take too long to realize we would need to:   Define a separator for the values in the list. Take care of certain exceptions, such as when the Text Input control is empty. Deal with blank spaces that could appear at the beginning and/or at the end of typed zip code values. Deal with mixed upper and lower case characters to support all types of zip codes around the world that could contain letters. Refine our requirements – for example: what happens if an incomplete zip code is typed, such as 234? Should we select all zip codes that contain 234 or should it be a full match? What about the opposite – user types 12345-678 and the table contains 12345 – should they match?   Here is what we have decided:   Using comma as the separator would work just fine, as comma is not a character used in zip codes. If the Text Input control is left empty, then it should have the same behavior as Visual Analytics: no filter is applied. Leading and trailing blanks for each of the values should be ignored, as they are not part of the zip code values being searched. We would standardize in upper case characters only. Typed values and values stored in the table should match entirely, so 12345 would not match 12345-678 and vice-versa.   To test if the input list of zip codes (a character parameter) is empty, we used the IsSet operator, and to ignore the filter if it’s empty, we used the expression 1 = 1 (which returns TRUE, as Visual Analytics doesn’t have a Boolean type):   IF ( 'Parameter'p IsSet ) RETURN (<expression>) ELSE ( 1 = 1 )   Now we needed to work on the <expression>. To remove blanks, we used the RemoveBlanks function. We could have used the RemoveChars function instead, which is more generic and would work for other types of characters. To standardize the input, we applied the UpCase function:   UpCase(RemoveBlanks('Parameter'p, _All_))    Our <expression> then became:   UpCase(RemoveBlanks('Parameter'p, _All_)) Contains UpCase(RemoveBlanks('ZipCode'n, _All_))   Observe that the same transformations were applied to both sides, just to make sure we are comparing apples to apples.   For the full match of individual values, the operator Contains naturally takes care of cases where the user types 12345 on the list of values and the table contains a zip code like 12345-678 (it’s a no match as defined), but the opposite is not true:   “zipcodeA,12345,zipcodeB” Contains “12345-678” --> returns FALSE (not a match): correct “zipcodeA,12345-678,zipcodeB” Contains “12345” --> returns TRUE (a match): incorrect   If the user types 12345-678 in the Text Input control, it would match zip code 12345, and potentially any other subset of the input value, such as those bad zip codes:   12345- 123 45-6 -678 - 7 etc…   To guarantee the full match of what was being typed, we had to search for the entire value, and an easy way to do that was to include characters to mark the beginning and the end of the values in the search string. Well, it turned out that by definition, all values were already separated by comma in the input text, so it naturally became the character to indicate start and end of values. All we had to do was to include the separator in the expression, by concatenating a comma before and after the zip codes stored in the table, using the operator Concatenate twice:   Concatenate( ',', Concatenate( UpCase(RemoveBlanks('ZipCode'n, _All_)), ',' ) )   After concatenating comma before and after:   “zipcodeA,12345, zipcodeB” Contains “,12345-678,” --> returns FALSE (not a match): correct “zipcodeA,12345-678, zipcodeB” Contains “,12345,” --> returns FALSE (not a match): correct   This seemed all we had to do, except for one thing: if user types 12345-678 as the first or the last element on the search list, or if it’s the only element in the list, it will not contain the separator character, and valid zip code values will not be found:   “12345-678,zipcodeB” Contains “,12345-678,” --> returns FALSE (not a match): incorrect “zipcodeA,12345-678” Contains “,12345-678,” --> returns FALSE (not a match): incorrect “12345-678” Contains “,12345-678,” --> returns FALSE (not a match): incorrect   To fix that we needed to append a comma at the beginning and at the end of the entire typed input parameter string, exactly as we did before with the zip codes stored in the table:   Concatenate( ',', Concatenate( UpCase(RemoveBlanks('Parameter'p, _All_)), ',' ) )   After we did that:   “,12345-678,zipcodeB,” Contains “,12345-678,” --> returns TRUE (a match): correct “,zipcodeA,12345-678,” Contains “,12345-678,” --> returns TRUE (a match): correct “,12345-678,” Contains “,12345-678,” --> returns TRUE (a match): correct   Putting everything together, this is how the filter expression looked like (before Viya 2023.06 syntax):   IF ( 'Parameter'p IsSet ) RETURN ( Concatenate(',', Concatenate(UpCase(RemoveBlanks('Parameter'p, _All_)), ',')) Contains Concatenate(',', Concatenate(UpCase(RemoveBlanks('ZipCode'n, _All_)), ',')) ) ELSE ( 1 = 1 )    Same expression starting with Viya 2023.06 syntax (basically IsSet and Contains are functions instead of operators and the key work _All_ no longer has underscores):   IF IsSet('Parameter'p) RETURN Contains( Concatenate(',', Concatenate(UpCase(RemoveBlanks('Parameter'p, All)), ',')), Concatenate(',', Concatenate(UpCase(RemoveBlanks('ZipCode'n, All)), ',')) ) ELSE ( 1 = 1 )    In our case, zip codes were already stored as characters in the source table, but if that was not the case, we could simply apply the Format function to transform them into characters, and the rest of the expression would be the same.   Obviously, as long as the requirements remain the same, this expression can be leveraged for other use cases, such as searching for car makes.   Suppose that you have a bar chart, and you want to be able to type one or more vehicle Makes separated by comma in the Text Input control object at the top to filter the chart: Figure 01-Text input and bar chart objects Let’s take a look at how you would do that in three easy steps.   FIRST, you create a parameter with the following attributes:   Name: list of Makes separated by comma Type: Character Multiple values: unchecked Current value: empty Figure 02-Create character parameter Knowing that this parameter will carry multiple values, you may be attempted to check the box for multiple values, but a text input control object, where this parameter is going to be assigned to receive its value, does not accept multiple values – the object is a text box, and therefore it accepts only a single value, which is a string.   SECOND, you assign the parameter to the Text Input control object. Note that before Viya 2024.05, the parameter is assigned in the object’s Roles pane: Figure 03-Assign parameter to text input object prior to Viya 2024.05    Starting in Viya 2024.05, this assignment is done in the object’s Actions pane:  Figure 04-Assign parameter to text input object in Viya 2024.05 and beyond   THIRD, you add the advanced filter discussed previously into the bar chart’s Filters pane – remember that the name of the parameter and the data item being searched are different in this example.   Now let’s suppose you type “Toyota,,   ,    bmw,  hoNdA  ,Mercedes,Volks Wagen     , xyz, Nissan SUV, Land rover”, without the quotation marks, in the Text Input object and hit Enter – you must hit Enter for the value to be assigned to the parameter. Starting from the inner most operators/functions and moving outwards, this is what happens:   Remove all blanks (space characters) from the input parameter. You should get this: Toyota,,,bmw,hoNdA,Mercedes,VolksWagen,xyz,NissanSUV,Landrover Make everything upper case as a best practice for string comparison: TOYOTA,,,BMW,HONDA,MERCEDES,VOLKSWAGEN,XYZ,NISSANSUV,LANDROVER Concatenate a comma at the end. Note that comma is used as the separator. It must be a character or combination of characters that do not appear in the data being filtered - all possible values that Make can assume in this example. This is what you have so far: TOYOTA,,,BMW,HONDA,MERCEDES,VOLKSWAGEN,XYZ,NISSANSUV,LANDROVER, Concatenate a comma (or the other elected separator) at the beginning. Let’s call this result the standardized parameter: ,TOYOTA,,,BMW,HONDA,MERCEDES,VOLKSWAGEN,XYZ,NISSANSUV,LANDROVER, Repeat the same steps for the Make column to create the standardized Make, so you can compare apples to apples. For example, if the value of the Make being compared is Land Rover, it would become: ,LANDROVER, Check if the standardized parameter contains the searched standardized Make. Only do all of that if the parameter is set. If it’s not set, return true, meaning all Make values should be displayed, which is the default in SAS Visual Analytics. Figure 05-Results  Note that Mercedes did not show up in the bar chart because it’s stored as Mercedes-Benz in the table. In another hand, Nissan is a valid car make but Nissan SUV is not, so it was not returned. Also, because the implemented filter is not case sensitive, Honda was displayed as expected, even if typed with mixed casing in a strange way. Because spaces are eliminated, mistakenly typed Volks Wagen (two words), matched the make Volkswagen. Non-existing makes such as xyz, empty values, and extra blank spaces didn’t affect the results.   Please, let me know in the comments what you think about this solution and how or for what purpose you have used it. If you have Viya release 2020.1.1 (December 2020) or above, I’m particularly interested in knowing why you had to adopt this solution instead the List control object with search capabilities available out of the box.         ... View more

If you have ever implemented a DDC that executes Viya Jobs, you certainly had to inspect the SAS log for clues about errors that might have happened, especially during the development phase, and even though you can ask the Viya Job to send back the SAS log, this most likely meant you had to modify portion of the JavaScript code that calls the job to set the parameter _debug=log, and in some cases the portion of the code that interprets the results received from the job as well.   When I first published the series of articles about integration of VA (SAS Visual Analytics) and Viya Jobs using DDC (Data-Driven Content object in VA), the third publication and the two subsequent use case examples used Viya Jobs that didn’t produce any visible output, but instead, they created tables that were loaded in CAS (memory). Because there was no visible output, the job ended up returning a small JSON structure to the DDC to inform the status of the SAS code execution, so after every SAS statement that could generate an error, we would call the macro function %stopOnError. This macro function does exactly what is says: it checks for errors in a few macro variables and if the error is found, it sends back to the DDC a small JSON message and aborts the job execution:   Figure 01-Example of old error message   Even though it works, there are some limitations on this approach: You have to modify the SAS code to call the macro function after every step that may throw an error. The error message, without a broader context or the exact location where the error occurred, may not be sufficient, which would still require to modify the code to call the job with _debug=log. It only worked for Viya Jobs that didn’t send any content back to the DDC, such as SAS ODS (Output Delivery System) output, files, etc. If any other content had to be returned from the job to the DDC, then we could not send the JSON message back. Well, it turns out that a Viya Job can return a list of output files in a JSON structure, and by leveraging that capability, we can present a new template of DDC + Viya Job that is a lot more flexible to return any content that you need, including the complete SAS log, while keeping the SAS code simple.   The advantages are: You don’t need to modify the SAS code by adding macro function calls in strategic locations to capture eventual errors. Any error in the SAS code is readily available in the SAS log file that can be retrieved and presented. It works with any kind of Viya Job, independently of the number and type of output it generates. You ultimately decide what to do with the output files: display in the web client (such as VA), download to your local computer, display in the browser’s console, etc.   Some disadvantages are: The output files generated with the SAS code must be stored in a location where it can be retrieved, and that normally includes content generated with ODS output. That means you will need extra ODS statements in the SAS code to store the ODS output in files. You can no longer send content to the DDC, such as ODS output that the browser understands, and hope the browser will automatically display it. You now control what to do with the different output files, which could also be an advantage (see #4 above) – it depends on how you look at it. You need to use JavaScript to post-process the JSON data and appropriately handle the list of output files, to give them their final destination. Because a DDC is implemented as an HTML file, and a Viya Job can have a form (HTML file) associated with it, whenever there is a DDC that calls a job, I like to deploy the DDC as the job form and keep them together. The key difference between a DDC that calls a job and a job form, in terms of functionality, is that the DDC communicates with VA by sending and receiving messages with data, parameters, selections, etc. Everything else is the same. A DDC could for example receive data from VA and upload that data as a table to be processed by the Viya Job, and the job could load an output table in CAS, so other VA objects could visualize its content, like seen in this series of articles. To keep the focus on the key points of this example, which is creating and handling multiple output files that the Viya Job creates, we will not be using a DDC. Instead, we will be using a simple job form with a text box to capture the name of the input table and a button to submit the job. Again, the job form could easily be transformed in a DDC and integrate with VA.   The example explored here produces HTML and PDF output. The HTML is to be displayed and the PDF is to be downloaded, just to demonstrate that you are now in control of what to do with what the job produces as output: Viya Job Checks for expected parameter (input table in the format library.table) and returns an application error if something is wrong. Produces the HTML and PDF output files (proc print of 10 first observations). Job form (remember that it could be a DDC) Acquires the input parameter (table name as library.table) and calls the Viya Job. Checks the job output for errors.       If error, displays log information in the browser’s developer console       If ok, processes the job output files (displays HTML and downloads PDF)   The Viya Job Code   All files generated by the job to be returned to the job form (DDC) must be saved in the Viya Content Server via a special filename statement: Figure 02-Filename statement     The name is important because this is how the job form (DDC) identifies this file in the JSON structure that it receives from the job, as you will see later.   Because the job can send back multiple files, I have adapted the old macro function %stopOnError (it’s ok if this is the first time you are hearing about this macro function) to send messages at the application level in a file named “_appout.json”. The macro was renamed to %sendApplicationMsg and is now used to inform errors that the application can identify, and abort the job execution if something is wrong. For example, we can use this macro in case an expected parameter was not passed to the job, or if a table doesn’t exist, etc. At the end, if the job runs successfully, you can also use that same macro function to inform the job form (DDC) that the job has reached the end and finished with success. Differently from the macro %stopOnError that had to be called multiple times in the SAS code, %sendApplicationMsg is only called if needed, according to the application logic. More precisely, in our example this macro function is used whenever a job input parameter is missing or the input table doesn’t exist. If you are wondering, we have a better mechanism to get information about unforeseen errors that will be explained later. This is an example of application message written to “_appout.json”:   Figure 03-Example of new application error message   The other two files that the job creates are named “_webout.html” and “_webout.pdf”, containing respectively the HTML and PDF output content. Those output formats are obtained via ODS. If you inspect the code, you will see that all ODS outputs are closed at the beginning of the code, and just before the proc print that generates a table with the first 10 observations is executed, two ODS outputs are open, one for HTML5 and one for PDF.   How those files are sent back to the job form (DDC) in a JSON structure depends on how the job is called, and it will be discussed in the next two sections: Calling the Job and Processing the Job Output.   This is the entire SAS code for our example:   Figure 04-Viya Job code   The Job Form (DDC) Code Calling the Job   The URL of the job form (DDC) is the URL of the Viya Job, with the parameter _action=form, which we normally add in the job properties, so we don’t have to pass that parameter all the time, keeping the job URL simpler. When the job form (DDC) calls the job, we would normally overwrite _action=form by explicitly passing _action=execute. But when we want the job to return a list of output files in JSON format, we must call the job with _action=json. Making a call to the job with _action set to json is the first key change.   Figure 05-Calling Viya Job with HTML form and _action=json   This is an example of JSON with output files that you get when the job is called with _action=json:   Figure 06-Example of JSON response with list of job output files   Except for the log file, which name is automatically created for you, the other names are defined by you, and they are set in the SAS code with the filename statement.   There are two other parameters passed to the job that we need to highlight:    _resultfile: By default, only file names matching _webout.* are included in the JSON output. If you want anything else to be included, you need to inform in this parameter. Wildcards “*” and “?” are accepted and multiple values can be separated by “,” (see documentation for more details). if _resultfile is such that it allows the SAS log to be included, a file named <SYS_COMPUTE_JOB_ID>.log is automatically added to the output, where <SYS_COMPUTE_JOB_ID> is the job ID and matches a macro variable with the same name in the job code. The *.log file is in JSON format and it contains more than just the SAS log lines, therefore a bit of post processing is needed. The extra information can be used to color code the log, in case you want to display the log in HTML format for example. _omittextlog: The default value of _omittextlog is true, but if it’s set to false, an additional SAS log file named <SYS_COMPUTE_JOB_ID>.log.txt is also automatically included in the output, as long as _resultfile permits. The *.log.txt file only contains the SAS log lines in text format and it’s easier to consume. We are not generating this file in our example.   Lastly, the header of the HTTP request must have “Accept” set to “application/json”, because this is what the Viya Job returns with _action=json.   In our example, all those steps are performed in the callJob() function (highlighted in yellow).   Figure 07-JavaScript function callJob()     Processing the Job Output   Once the job finishes and returns execution to the job form (DDC), you need to inspect its response as a JSON. But what if there was an error when the job was executing that you were not able to anticipate and inform it through a call to the macro function %sendApplicationMsg? Well, calling the job with _action=json has another advantage: if there is an error in the SAS code execution, a special JSON output is automatically sent back, according to the documentation:   Figure 08-Example of JSON response when there is an error in the SAS code   As you can see, the attributes in this JSON can be used to not only detect there was an error, for example, by inspecting the “errorCode”, but also to have additional insights on the error, including the entire SAS log. So once the job returns execution to the job form (DDC) and an error is detected, you can dump the SAS log in the browser’s developer console for easy troubleshooting (please see code highlighted in purple above). Obviously, you can skip this step after development is concluded and the solution is deployed in production.   Note: in Viya3.5, the response received when there is an error executing the SAS code is slightly different and is documented here. It contains not only the error message, but also the JSON structure with the result files that were specified in the _resultfile parameter. This is important because instead of reading the response as a JSON, you need to read it as a text and then parse its components into their own JSON objects.   If there was no error, the JSON structure received is actually the job output, with URI to the output files in it, as specified in the _resultfile parameter, so the next step is to inspect the content of the file we called _appout.json to see if there was no error at the application level. Remember that application errors are those that you generated when validating input parameters and are different from unexpected execution errors. If there was no application error, then you should access the output files URI, extract their content, and give them the proper destination: display the HTML and download the PDF. In our example, these are all performed in the Submit() function:   Figure 09-JavaScript function Submit()     To download the PDF file, you call the function downloadFile(). This function uses JavaScript to create a <a> element on the fly and simulate a user clicking in its hyperlink, which points to the PDF file’s URI.   Figure 10-JavaScript function downloadFile()     Deploying and Running the Example   The complete SAS code and job form are attached in a zip file. Once you add them to a Viya Job, you can run it directly from SAS Studio or SAS Job Execution web applications. Remember to set the parameter _action=form in the Viya Job properties.   The following two videos show how to create and run the job with the provided files in SAS Job Execution web application, and how to easily check the SAS log in the browser’s developer console (Ctrl+Shift+i) when there is an error in the SAS code.     ... View more