The latest release of SAS Visual Analytics 8.4 is out with a lot of new features and product enhancements. But this one in particular, the SAS Visual Analytics SDK is my most favorite. With SAS Visual Analytics (VA) SDK, organizations can leverage the openness of SAS platform and SAS Visual Analytics and build customized and personalized web applications, thereby providing a deep linking for embedded insights generated by SAS Visual Analytics into their own web applications and custom portals.     So, what is SAS Visual Analytics SDK?   SDK stands for “Software Development Kit”, which is a great way to think about it — a kit. Think about putting together a model car or plane. When constructing this model, a whole kit of items is needed, including the kit pieces themselves, the tools needed to put them together, assembly instructions, and so forth. An SDK or devkit functions in much the same way, providing a set of tools, libraries, relevant documentation, code samples, processes, and or guides that allow developers to create software applications on a specific platform.    The SAS Visual Analytics SDK is a collection of JavaScript libraries that web developers can use to embed SAS Visual Analytics insights within their software applications and customized web pages. Using the power of SAS Viya server as the backbone, the SDK will give businesses and developers the flexibility to build and customize an organization's user experience that meet their specific design/brand requirements and provide a wide range of SAS Viya insight capabilities that can be embedded into and delivered as part of their applications.   Ways to Integrate   SAS Visual Analytics SDK allows you to embed an entire VA report or individual report objects within your customized application or portal web page. The content you embed will be fully interactive and any interactions defined on those objects will remain functional, even when embedding individual objects including filter and brush interactions.   The ultimate goal of VA SDK is to bring embedded insights to the market. The intention is that objects are still designed in Visual Analytics, with options and interactions defined there in Visual Analytics. The VA SDK allows us flexibility to display our content into other third party or customer proprietary portals in a light weight manner.   This functionality is delivered as the va-report-components JavaScript library. With this library we could:   -: embed entire report or page with the <sas-report>custom HTML element tag  -: embed a single object with the <sas-report-object> element tag   -: customize your own HTML content based on data and actions that are driven from SAS Visual Analytics   For the official documentation on the various ways you can integrate refer to: developer.sas.com     Getting Started   In order to get started with using the VA SDK, you will need the following VA SDK libraries: either downloaded or pointing to a mirrored Content Delivery Network (CDN) location Access to a VA 8.4 on Viya 3.4 deployment with additional configurations for Cross-Origin Resource Sharing (CORS) and guest access A Web Server that can host your portal page (html) consuming the embedded insights from VA 8.4 report\report objects These pieces could be all on the same server or on totally different servers on different networks.  Now let us look at each of these items in greater detail below.    SAS Visual Analytics SDK libraries    VA SDK libraries are published to npm (node package manager) where it can be downloaded and installed.   Since npm packages are mirrored by many Content Delivery Networks (CDN’s)you could also point to a CDN site directly instead of downloading the libraries. Refer to the Getting Started section of the developer.sas.com for installation instructions.    SAS Viya configuration for SAS Visual Analytics SDK    The following SAS Viya configuration needs to exist for VA SDK to function properly:   Enabling Guest Access  Connection to reports and objects in a report embedded into a third party portal web page can be made using the Guest access. Currently the va-report-components library only connects as the Guest user. This requires that the Viya system be setup to enable Guest, and that the Guest user have access to the appropriate report content and data. See Enabling Guest Access for more information. Future updates of SAS Visual Analytics SDK will support authentication and single sign on.   Enabling CORS  By default, your Viya deployment is not setup to allow access to REST api endpoints from different domains. This is needed in order to connect to Viya from the domain that is hosting your html page. This domain needs to be added to the allowOrigins property in Viya deployment's CORS configuration. See developer.sas.com and SAS Help Center for more information on enabling CORS.    Guest Access to reports, data and libraries The guest user should have access to reports, and the underlying data that the report is based upon.   In my example below, I have enabled guest access on my Viya server, created a folder called "Reports for guest users" and set the permissions on the folder such that guests will get to Read and Read (convey) on all the reports in this folder. Further I moved the reports I wanted to share with my guests to this folder making sure the guests also have access to the underlying data and the CAS library that holds this data. It is very important to note here that your guest users not only needs access to the reports but also need access to the data that the report is based upon.           Permissions on "Reports for guest users" folder:     Creating a web server   Mac OS You will also need a web server that can host your portal page (html) consuming the embedded insights from VA 8.4 report\report objects. Your Mac comes with a built in powerful web server for free. Anyone will be able to access the website stored on your mac and your website will be available any time your mac is turned on and connected to the internet. Versions of OS X prior to El Capitan, Yosemite, Mavericks, Mountain Lion and Mavericks can simply turn on “Web Sharing”, but from 10.8, 10.9, 10.10, and 10.11 onward you’ll need start the apache web server using the following command in a terminal. Enter the admin password when requested.                     sudo apachectl start     Launch Safari, Chrome, or Firefox on the browser on your local machine and navigate to “http://127.0.0.1” to verify the server is running, you will see an “It Works!” message.   The default configuration files for the apache web server is under /etc/apache2/httpd.conf And the default document root directory for the files as specified in the httpd.conf file is under            /Library/Webserver/Documents/     This location can be changed, but will require a restart of apachectl            sudo apachectl restart    The index.html file that servers the "It Works!" message resides here.  We will be creating our example VASDK.html report example page in this location.     How to find your Mac's external ipaddress: Now you will need to know your Mac's external network address so you can access it from a remote computer on your network. There are many ways you could find this:   On your Mac, click on System Preferences->Network You can select the connection that you are connected to and that should give you the ip address, Alternatively - you can go to System Preferences -> Sharing and click on Remote login and that should give you the full hostname for your machine.   Alternatively you can also use the following ifconfig command line option to get the external ipaddress for yor Mac:           ifconfig | grep "inet " | grep -v 127.0.0.1   Refer to the following article: Find your IP Address on a Mac for step by step instructions with pictures.    At this point you could do a http://<youripaddress>:80 from a remote computer on your network and you should be able to see the "It Works!" message.     Windows 10 Similarly you can follow the link below to setup a IIS web server on Windows 10: Creating web server on windows 10   VA SDK Report Example using embedded insights   Find attached a zip file that contains VASDK.html, article.css and a logo.png file. Copy the VASDK.html as is and place it in the /Libraries/WebServer/Documents location of your Mac machine.   Copy the accompanying article.css and the logo.png file as well to the same location.   Make sure these files have Read permissions set for all users.   Read thru the VASDK.html to get familiarized with the various sections in there.    Now let us look at some of the tags in the VASDK.html      Loading the VA SDK Library   The following script tag in the html loads the VA SDK library. Notice in this case we are loading from a mirrored CDN location. In case you decide to download the library, the path will be modified accordingly.  This is just an example. Please refer to the official developer page to get the latest tag for the library.   <script async src="https://unpkg.com/@sassoftware/va-report-components@0.1/dist/umd/va-report-components.js">   </script>   Embed a VA report or report object   To add a Visual Analytics report\report object to the page you will need to know two pieces of information: Viya url: The root url of the Viya system on which you have Visual Analytics. In my example it is  https://birdsgf01.unx.sas.com Report uri: The uri of the report to embed. This value takes the form of /reports/reports/${UNIQUE_ID} and can be found in the link url of the Visual Analytics share report dialog, as shown here. Once the report link has been copied to your clipboard, use the custom tag generator in Getting Started page to generate the <sas-report> tag.   <sas-report authenticationType="guest" url="https://birdsgf01.unx.sas.com" reportUri="/reports/reports/c3c6befb-3981-4c9e-b011-7dc11dec5e37" hideNavigation ></sas-report> Object id: It is also possible to pick out individual objects from a Visual Analytics report and add them to your page. To do this you will need to know one additional piece of information: The unique identifier of the object. Like the report uri, this can also be found in the link url of the Visual Analytics share object dialog, as shown here. Once the object link has been copied to your clipboard, use the custom tag generator in Getting Started page to generate the <sas-report-object> tag.     <sas-report-object authenticationType="guest" url="https://birdsgf01.unx.sas.com" reportUri="/reports/reports/698882d7-e119-45fc-a0f4-fc86f41392d1" objectName="ve27663" ></sas-report-object>   The example VASDK.html has a banner and a placeholder for four VA report objects. The title for every section has a link to view the pdf version of the report. In order for this to work make sure you save the pdf version of the report in the location where you save the VASDK.html   The HTML tag to add additional objects is as follows. Use this example below to modify the html to match values from your own environment. The goal of this example is to give you some basic files to get your own example going so that you can explore and play around with this functionality.        <div class="visual-container"> <div class="object-header"> <h2><a href="./WaterConsumptionMonitorReport.pdf">Top Water Consumers Dashboard</a></h2> <h3>Click on the link above to get a pdf version of the full report</h3> </div> <sas-report-object authenticationType="guest" url="https://birdsgf01.unx.sas.com" reportUri="/reports/reports/c4e55896-bd3b-4901-95d7-09157925e440" objectName="ve128" ></sas-report-object> </div>   Since my first 2 side by side objects are from the same report, when you interact with one object the interactions will take effect in the other object as well as shown below:         As you can see the SAS Visual Analytics SDK is a very powerful tool allowing users the flexibiliy to display and customize SAS Viya report content into other third party or customer proprietary portals and applications. Feel free to let us know if you have any questions or suggestions in the comments section below.    Also, refer to the SAS Visual Analytics 8.4 and the new SAS Visual Analytics SDK are now available! link for more details and documentation links.    Additional Resources: What's new in SAS Visual Analytics 8.4 SAS Visual Analytics SDK  Getting Started with SAS Visual Anlytics SDK Node Package Manager Installation VA SDK on github     ... View more

This post is intended to demonstrate the steps involved in building a SAS Viya programming only docker recipe and deploying that into a docker container. With the current release, any SAS customer who licenses SAS Viya, will have the ability to download the SAS Viya Programming-only SMP container offering with SAS Studio and run it in a docker container. This offering is called the Programming-only analytics container. It will not be a full-fledged SAS Viya deployment.   SAS Viya Programming-only analytics container offering with SAS Studio will have a recipe for how to efficiently build and deploy a docker container. This is the most flexible option as it will allow customers to layer their preferred OS and can pick and choose the components they would like to use to build their images and deploy the containers. To start, access the open source project on GitHub.   Docker You can run docker in a windows, linux or a Mac environment. Make sure you have docker downloaded and installed and docker is started and running. If you are new to docker here is a good place to start: https://docs.docker.com/get-started/   Getting Started   Before you can install the SAS Viya Programming-only analytics container, you will need a valid SAS Viya Software Order. It does not matter what you have in your order, you could have VA, VS, VDMML or any other pieces: this recipe will only build and deploy the programming only environment in a container.   Download or clone the GitHub repository to your machine in a folder called /containers/poac.  Notice that the repo creates a folder called sas-container-recipes-master. Change directory into sas-container-recipes-master. Notice the various folders in this directory, This will be your root folder. From the root of  this folder we will be using the build.sh script to create the various docker recipes. Copy the SAS_Viya_deployment_data.zip that came with your SAS Software order to the viya-programming/viya-single-container directory.   What is a SAS Viya Programming-only analytics container recipe?   A docker recipe allows you to build a set of docker images that represent a SAS Viya 3.4 usecase. A recipe provides end-to-end scenarios for exotic or otherwise advanced use-cases. Most users will not have a need for all of these, they can pick and chose what they want in their image. Recipes allow you to chose and build the images as you want. A base viya-single-container image is the starting point. Running the build.sh script will give you a SAS Viya 3.4 programming image. To this image you could add the addons to create custom images of your choice.   I guess you could use the analogy of a cake layering recipe here. Baking is not my forte so I may be using this poorly, but basically you select the layers you want in your cake, and “cook” them from the bottom layer up, building the cake layer by layer. A Programming-only docker recipe is just that. You build the base viya-single-container image and build it up by adding the other layers to it.   For example:   auth-demo:  The auth-demo project will provide the ability to create a default user. It you specify nothing, then that user is “sasdemo”. If one provides specifics then the specific user that is created is whatever was provided. This would be useful for someone who would want to get a SAS Viya image up and running quickly and did not want to tye the container into LDAP, then one could use the auth-demo as a way to create a quick user that would be the CAS Admin user and the user for Jupyter.   auth-sssd: This provides a structure on how to hook in LDAP configuration that uses sssd and PAM   ide_jupyter_python3: This project allows you to add Jupyter Notebeook with Python3 to a SAS Viya programming image access-xxxx addons: these projects allows you to configure the various  SAS Access interfaces to the respective databases. In this blog post we will be using the build.sh that exists in the root of the project to quickly build a set of docker recipes (images) that represent a SAS Viya 3.4 usecase.   Recipe 1: creating the viya-single-container (svp base image) with a default sasdemo user This recipe will give you a viya-single-container image with svc-auth -demo: The auth-demo project will provide the ability to create a default user. It you specify nothing, then that user is “sasdemo”. If one provides specifics then the specific user that is created is whatever was provided. To build this image you would issue the following command.   ./build.sh addons/auth-demo   The build step will take a while (about 30 min or so) while it downloads all the packages and installs. It will print what is happening to the console as well as to a file named build_sas_container.log. It will install ansible and other pieces that are needed in order for ansible to run. Once the playbook is done running, ansible is removed as it is no longer needed.   Once the image is built you could issue a “docker images” command to see the list of images that has been built.  A base centos image gets built by default.     The svc-auth-demo project will provide the ability to create a default user. It you specify nothing during the docker run, then that user is “sasdemo”.  This user is also the CAS Administrator.   To run this container issue the following command:   docker run –detach –rm –env CASENV_CAS_VIRTUAL_HOST=birdsgf23.unx.sas.com –env CASENV_CAS_VIRTUAL_PORT=8082 –publish-all –publish 8082:80 –name svc-auth-demo –hostname svc-auth-demo svc-auth-demo   You could provide another user instead of “sasdemo” by specifying the following arguments:   --env CASENV_ADMIN_USER=foobar --env ADMIN_USER_PWD=barbar   If you do that then the default user that would be created is “foobar” with a password of “barbar”. The “sasdemo” user will not be created in this case. Only the user “foobar” is created. For our example let us use these extra options. A “docker ps” command shall tell you that this image is running on port 8082     If you want to see all the processes running in this container you can issue the following command:   docker exec -it svc-auth-demo ps -ef     You could also issue the following command to check the status of the Viya services  running in this container:   docker exec --interactive --tty svc-auth-demo /etc/init.d/sas-viya-all-services status     At this point you could bring up SAS Studio with the following url:   http://<yourservername>:8082/SASStudio   You should be able to log in as user “foobar” and a password of “barbar” Also log in to the CAS Administrator console as “foobar”   http://<yourservername>:8082/cas-shared-default-http/   and notice that “foobar” is the CAS Administrator.     Recipe 2: configure LDAP that uses sssd and PAM This project allows us to create an image with sssd support and allowing a user to set the ADMIN_USER to a LDAP user. You will need a sssd.conf file with all the custom ldap configurations placed in the auth-sssd folder. You could enable encryption by providing a sssd.cert file and placing that in the auth-sssd location.   In order to build this image simply issue the following command:     ./build.sh addons/auth-sssd   This image will only build the base image with auth-sssd  This may be desirable for security conscious customers that may decide to have only ldap users and not have local host account in the container. They may also choose to have one of the LDAP accounts to be the CAS Admininstrator.   To run the svc-auth-sssd container with a ldap user as a CAS Administrator, issue the following command. I want to make myself as an admin so I pass in my ldap userid as the CAS Admin in the option below:   docker run –detach –rm  –env CASENV_ADMIN_USER=mevenk   –env CASENV_CAS_VIRTUAL_HOST=birdsgf23.unx.sas.com –env CASENV_CAS_VIRTUAL_PORT=8083 –publish-all –publish 8083:80 –name svp_auth_sssd –hostname svp_auth_sssd svp_auth_sssd     Recipe 3: I want it all! I am a user who wants it all. I want a base SAS Viya programming image with the default user, sssd and jupyter-python. To accomplish this I would issue a build command as follows:   ./build.sh addons/auth-demo addons/auth-sssd addons/ide-jupyter-python3   The build.sh was setup so that you can pass in the collection of things you want to build. Running the above would create the following 4 images:  viya-single-container, svc-auth-demo, svc-auth-sssd and svc-ide-jupyter-python3, where each one is based on the previous one.   Mae sure you have the custom sssd.conf file and the sssd.cert file copied in the auth-sssd folder before you begin building.     Since I want my image to have a collection of all of these things – I would only run my last container: svc-ide-jupyter-python3 Further, I would like my CAS Administrator to be a host user named “foobar” with a password of “barbar” .   Here is how my docker run command would look like:   docker run –detach –rm –env CASENV_ADMIN_USER=foobar –env ADMIN_USER_PWD=barbar –env CASENV_CAS_VIRTUAL_HOST=birdsgf23.unx.sas.com –env CASENV_CAS_VIRTUAL_PORT=8084 –publish-all –publish 8084:80 –name svc-ide-jupyter-python3 –hostname svc-ide-jupyter-python3 svc-ide-jupyter-python3   A “docker ps” should tell me that the svc-ide-jupyter-python3 image is running on port 8084      Testing with SAS Studio   Bring up SAS Studio: http://<yourservername>:8084/SASStudio/   Since we have sssd setup, you can log in as your ldap userid and password. Create a cas connection and create a new cas session. For your cashost you can use localhost. But in the steps below I will also show you how to get the host name of the cashost.     CAS Administrator Console   Bring up CAS Administrator Console: http://<yourservername>:8084/cas-shared-default-http/ Log in as the CAS Administrator “foobar” and a password of “barbar”. This user was created in the previous step during the docker run step. You will notice that the above code I ran in Studio as myself (mevenk) started a couple of cas sessions.     And notice that the CAS Administrator is “foobar”     In order to get the hostname for the cas host, click on “Runtime Environment” and scroll down to host. The cashost in this case is svc-ide-jupyter-python3     In defining the cashost, you can either use localhost or the cas hostname such as below   options cashost="svc-ide-jupyter-python3" casport=5570; cas;   Testing SAS Kernel with Jupyter   Bring up Jupyter notebook: http://<yourservername>:8084/Jupyter                      Open the SAS kernel in Jupyter     The following piece of code will run the sas program in Jupyter Notebook.   ods html5 style=statistical; ods graphics / width=500 height=400; proc means data=sashelp.cars; run; proc sgplot data=sashelp.cars; histogram msrp; density msrp; run;       Testing python3 kernel with Jupyter   Now open a python3 kernel in jupyter and run the following cas action:   import swat s = swat.CAS("svc-ide-jupyter-python3", 5570) s.builtins.serverStatus() import pandas as pd import matplotlib.pyplot as plt from IPython.core.display import display, HTML %matplotlib inline titanic3 = s.CASTable("titanic3", replace=True); s.upload_file('http://biostat.mc.vanderbilt.edu/wiki/pub/Main/DataSets/titanic3.csv', casout=titanic3) type(titanic3) titanic3.columnInfo()       For more about this example and the code – refer to the following documentation.   Note: The authinfo.txt needs to be in the users home directory if you want Jupyter to communicate with SAS *without* having to code user name and password in the SAS code. Jupyter runs as a specific user (Trusted)  and thus taking you in that way. If you want a sense of security, you can provide the JUPYTER_TOKEN at “docker run” time which will make a user provide the token when going to the URL. By default this value is empty. More about this in my next blog.   At this point if you were to see the use sessions in the CAS Admininstrator  Console – you should see the cas user sessions we created above in SAS Studio and python3 kernel in Jupyter.     Cleaning up and Starting again    The best part about containers is that they are lightweight and easy to use that I can cleanup and start all over again in minimal amount of time. The build scripts do not take long to build and starting a container is almost instantaneous. Here are some docker commands that you will be using:   # gives you a list of all docker images\containers docker images # gives you a list of all running docker containers docker ps # will stop all running docker containers docker stop $(docker ps -a -q) # will remove a particular docker image # a "dockar images" command will get you the ImageID docker image rm <ImageID>   Closing Remarks   I hope you will all give this a try and post your comments here. Here are some additional reading, that you will find interesting. SAS Communities: Running SAS Analytics in a Docker container SAS Communities: You can have your cake and eat it too SAS for Containers Docker Toolkit for Data Scientists ... View more

Debugging performance issues across distributed systems is a challenging task and it will need a step by step approach to identify the cause of the bottleneck.  When it comes to performance, there are a myriad of factors involved with each one contributing in its own way to the overall problem. Sometimes, it’s like finding a needle in a haystack. Isolating the various factors is the key.   This document explains the step by step procedure used to debug report level performance issues in Visual Analytics (VA) and Visual Analytics Viewer (Viewer) application. The tests were conducted in SAS Visual Analytics 8.2 running in the SAS Viya 3.3 environment, but is also valid for the next VA release namely: SAS Visual Analytics 8.3 running in the SAS Viya 3.4 environment. We will go through an example scenario of a complex report with many interactions and multiple filters.    Before we begin – make sure the network connectivity, network latency, sizing, firewall and other items affecting performance are tuned to an optimal level and performance for these areas are acceptable. We want to make sure the issues we see in VA are not related to any of the reasons mentioned above. In this article, we will only be focusing on the report level performance issues.   Baseline # 1: simple report behavior The very first thing to do before we start our performance assessment is to get some baseline numbers. The first baseline you would want, is to assess the performance for a simple basic report with just a list table. Create a test report with a list table using a simple data set such as cars or class from the sample data. Using the "File -> Open" dialog in VA, open the report. Note down the timings. Now use the main menu to switch into the Viewer application. Wait for any thumbnails to render, and then open the report: either by clicking on the thumbnail from the recents list, or from content navigator.  In our tests here at SAS, we experienced anywhere between 4-7 seconds when using a simple stopwatch timer. The tests were done in a 4 node distributed MPP enviroment with the class data that comes as part of the sashelp library. Our servers are optimally sized and are located in a data centre with very low network latency. If you are seeing an unusually large discrepancy, then further analysis should be done as to why this simple report is taking that amount of time. The 4-7 second timing that we noted, may be different in your environment depending on these and other factors.   This is your Baseline # 1 test. Once you have an acceptable performance with your simple test report, gauge the behavior of your problematic report and document what you see. In our usecase, we saw rendering times of about 4 min in Viewer for our problematic report. Which indicated to us that there was something odd going on with our report which needed further investigation.   Enable Logs for DEBUG While diagnosing a performance problem, we typically enable logging for Report Packages and Report Data right out of the gate. These two service logs will give a lot of valuable information. When opening a Report, the Report Viewer first generates and downloads a Report Package, with the report xml, css files, images, etc. – and then submits queries for each of the visuals on the page. In most cases, the Report Data service log is going to be the most interesting and valuable, as it includes a breakdown of the time spent on each query, and query times are typically the most time-consuming part of the process. But the Report Packages log provides a good overview and can be helpful in diagnosing delays related to image processing or other file retrievals. Significant time spent in the initial phase of the report open process (ie. before the report starts to render with individual spinners for each of the visuals) suggests a slowdown in report packaging. Note that dynamic prompts are resolved during this step, so heavy usage of dynamic prompts could increase this time.   In SAS Environment Manager, under Configuration Management, set the following properties:   Report Data Service: (set all at DEBUG)                com.sas.bidata com.sas.cas com.sas.reportcache com.sas.reportcommon com.sas.reportcommon.utils.debug.Time com.sas.reportdata   Report Package Service (set all at DEBUG)             com.sas.reportcache com.sas.reportcommon com.sas.reportcommon.utils.debug.Time com.sas.reportpackages   Baseline # 2 Next baseline you want to get is to compare the performance of your problematic report in VA vs Viewer, cached vs non-cached and maybe also Chrome vs Edge browser. VA and Viewer results should be comparable and should give similar results. High level steps are as follows (Detailed steps are down below): Start the test fresh by clearing the report data results cache Open the report in VA and capture the query times. Open it a second time in VA and capture the query times from that run (to see if we are getting benefits from the cache). Open in Viewer twice and capture the query times for each Capture the logs This will give us a high level picture of the overall performance, and we can evaluate the logs from here to see where to focus next.   Query Diagnostic Tool Query Diagnostics Tool is provided within the SAS Visual Analytics application. This tool can be used to investigate issues concerning the queries that SAS Visual Analytics generates for the CAS server             Open your report in SAS Visual Analytics.            Select any object in your report.            Press Ctrl+Alt+Q on your keyboard You will notice the Query Diagnostic Window pop up.    For more details on how to use the Query Diagnostic tool: refer to the following SAS note   Clear report data results cache In the Query Diagnostic tool window click on the trash can on the top right corner (number 4 in the picture above). Clicking on this icon clears the VA and Viewer report data results cache.    Note: You need to be logged in as an Administrator when attempting to clear cache. It will not work if you are not an administrator, and there is no indication in the tool telling you that it did not work.   Conduct the Baseline # 2 test Restart report-data service to get clean logs On a linux xterm: tail -f the report-data logs. This is imp. Log into VA If asked, say Yes to being an administrator In VA – Instead of opening a report, Create a New report. Clear report data results cache: Ctrl+Alt+ Q and then click on the Trash can In the report-data log – you should immediately see a message such as this:   2018-06-04 16:11:44.008 INFO 88684 --- [o-auto-1-exec-5] c.s.r.service.ReportDataService: mevenk(23fb06cf) [0e0f5fe807bc2aa4] Results Cache is cleared Close the Query Diagnostics window. Open the report in VA. Note: Report will open without cache. As the report is rendering, you will notice a lot of messages in the report-data log Note down the clock time when the rendering is complete In the report-data log you should see a query such as this when that report fully renders. This is the indication of the time the last VA query took to complete. VA does not give us a good record of the first and last query in the logs and hence having this information helps us to isolate the first run from the second run of opening the report in VA.   Total elapsed time (ms) for GetData - QUERY - query:b63da329-150c-43dd-99b7-36a279c8a90a_c45  - [dd124922] = 206621             Reopen the report in VA using the File menu option. This time report will open with cache. Start the Viewer test. From the main menu, go to View reports, open the report. Notice report will open without cache. Close the report Re-open the report again – Notice that this time report will open with cache. Stop the test Collect the report-data log and move it to your personal machine where you have Text Pad. Since report-data service was restarted before the test, we should have clean logs for the following tests and in the following order. It also helps to do the tests while no one else is on the system. This may not be possible in many cases but if we can keep the logs as clean as possible - we will be better able to understand where the root of the problem lies.     VA without cache (Total Time: 3.55 min) VA with cache (Total Time: 30 sec) Viewer without cache (Total Time: 3.58 min) Viewer with cache (Total Time: 38 sec)   Analysis of sas-report-data Logs Both VA queries and Viewer queries go through report-data service and publish information there. If you open a report in VA, we would expect to see log messages referencing 'GetData - QUERY' - and if you open a report in Viewer, we would expect to see log messages referencing 'GetData - PACKAGE' or 'GetData - REPORT' or more specifically - one 'GetData - PACKAGE' and lots of 'GetData - REPORT' In TextPad open your report-data log Do a search on “total elapsed time” and “Mark All” instances of the search Close the search window, Edit -> Copy Other -> Bookmarked lines Paste it in a new Textpad document and call it "queries.txt". This will give you a summary of the time it took to run the individual queries in VA and Viewer. As an example: the following sequence represents the typical requests you will see coming through when opening a report in VA:   Total elapsed time (ms) for GetData - QUERY - query:e783e390-67be-4bcc-bc69-42bfc001a9a4_c39 - [dd106441] = 534  Total elapsed time (ms) for GetData - QUERY - query:fb4bfc2e-4b9c-4976-a304-30a60e1d07ac_c42 - [dd128265] = 595 Total elapsed time (ms) for GetData - QUERY - query:695004cf-0b99-4065-9a6f-a6751eff97bf_c37 - [dd124922] = 2497   As an example: the following sequence represents the typical requests you will see coming through when opening a report in Viewer: Create Metadata Supplement, Get Package Data, and then a series of Get Report Data calls.      2018-05-30 21:59:36.205 --- Total elapsed time (ms) for CreateMetadataSupplement - custom = 2940 2018-05-30 21:59:38.284 --- Total elapsed time (ms) for GetData - PACKAGE - pkg:i_08232f9b-406e-4306-b4e0-5cc8a1ab5e53/nodata - [dd23, dd81] = 2024 2018-05-30 21:59:54.791 --- Total elapsed time (ms) for GetData - REPORT - uri:/reports/reports/510c5a9b-f842-41a3-be64-0e53619b250e - [dd81] = 836 2018-05-30 21:59:54.824 --- Total elapsed time (ms) for GetData - REPORT - uri:/reports/reports/510c5a9b-f842-41a3-be64-0e53619b250e - [dd23] = 869 2018-05-30 22:00:04.084 --- Total elapsed time (ms) for GetData - REPORT - uri:/reports/reports/510c5a9b-f842-41a3-be64-0e53619b250e - [dd81] = 1759 2018-05-30 22:00:04.137 --- Total elapsed time (ms) for GetData - REPORT - uri:/reports/reports/510c5a9b-f842-41a3-be64-0e53619b250e - [dd23] = 1632 2018-05-30 22:00:04.392 --- Total elapsed time (ms) for GetData - REPORT - uri:/reports/reports/510c5a9b-f842-41a3-be64-0e53619b250e - [dd81] = 1500   If any of the thumbnails required rendering when you opened a report in Viewer, you will see log messages from these calls as well. These would appear as pairs of CreateMetadataSupplement / GetData - PACKAGE, where the package data call is a ‘baseline’ package with a single dd reference. Something like this …       2018-05-30 21:59:36.205 --- Total elapsed time (ms) for CreateMetadataSupplement - custom = 1221 2018-05-30 21:59:38.284 --- Total elapsed time (ms) for GetData - PACKAGE - pkg:b_08232f9b-406e-4306-b4e0-5cc8a1ab5e53/baseline - [dd12] = 1234   'interactive' packages are generated by SAS Visual Analytics App (mobile) and include processing of all queries. 'nodata' packages are generated for the web viewer.  The first example below is indicative of a mobile request. The second is what we would expect to see from the Web Viewer.     2018-05-30 21:59:38.284 --- Total elapsed time (ms) for GetData - PACKAGE - pkg:i_08232f9b-406e-4306-b4e0-5cc8a1ab5e53/interactive - [dd23, dd81] = 2024 … vs … 2018-05-30 21:59:38.284 --- Total elapsed time (ms) for GetData - PACKAGE - pkg:n_08232f9b-406e-4306-b4e0-5cc8a1ab5e53/nodata - [dd23, dd81] = 2024   Analysis of VA queries in detail   Now let us look at the queries.txt file you created above. Identify and separate VA queries without cache and with cache. As I mentioned before, VA does not give us a good record of the first and last query from a report run, hence having the information about how long that last query took to run, helps us separate the VA queries from our tests.   Do the same with the Viewer queries. They are easy to separate as each Viewer report open starts with a clear GET DATA - PACKAGE. Add some comments for documentation. See below the example log of my last nine VA queries for my non cached tests below.   Non-cached VA queries Total elapsed time (ms) for GetData - QUERY - query:e783e390-67be-4bcc-bc69-42bfc001a9a4_c39 - [dd106441] = 534 Total elapsed time (ms) for GetData - QUERY - query:fb4bfc2e-4b9c-4976-a304-30a60e1d07ac_c42 - [dd128265] = 595 Total elapsed time (ms) for GetData - QUERY - query:695004cf-0b99-4065-9a6f-a6751eff97bf_c37 - [dd124922] = 2497 Total elapsed time (ms) for GetData - QUERY - query:fb4bfc2e-4b9c-4976-a304-30a60e1d07ac_c43 - [dd128229] = 2507 Total elapsed time (ms) for GetData - QUERY - query:695004cf-0b99-4065-9a6f-a6751eff97bf_c36 - [dd106578] = 4001 Total elapsed time (ms) for GetData - QUERY - query:695004cf-0b99-4065-9a6f-a6751eff97bf_c38 - [dd121960] = 4678 Total elapsed time (ms) for GetData - QUERY - query:695004cf-0b99-4065-9a6f-a6751eff97bf_c44 - [dd106578] = 73837 Total elapsed time (ms) for GetData - QUERY - query:695004cf-0b99-4065-9a6f-a6751eff97bf_c46 - [dd121960] = 73826 Total elapsed time (ms) for GetData - QUERY - query:695004cf-0b99-4065-9a6f-a6751eff97bf_c45 - [dd124922] = 209421 Cached VA Queries Total elapsed time (ms) for GetData - QUERY - query:6ff3c57f-20d1-413e-a943-219e14c2b1f8_c91 - [dd106441] = 96 Total elapsed time (ms) for GetData - QUERY - query:e69d9a2c-869a-45d1-b2f8-c4b995e63552_c86 - [dd128265] = 1351 Total elapsed time (ms) for GetData - QUERY - query:a66c3df3-279e-4031-86fb-736e104d5ad8_c89 - [dd124922] = 1194 Total elapsed time (ms) for GetData - QUERY - query:e69d9a2c-869a-45d1-b2f8-c4b995e63552_c88 - [dd128229] = 1636 Total elapsed time (ms) for GetData - QUERY - query:a66c3df3-279e-4031-86fb-736e104d5ad8_c87 - [dd106578] = 105 Total elapsed time (ms) for GetData - QUERY - query:a66c3df3-279e-4031-86fb-736e104d5ad8_c90 - [dd121960] = 1163 Total elapsed time (ms) for GetData - QUERY - query:a66c3df3-279e-4031-86fb-736e104d5ad8_c94 - [dd106578] = 404 Total elapsed time (ms) for GetData - QUERY - query:a66c3df3-279e-4031-86fb-736e104d5ad8_c96 - [dd121960] = 422 Total elapsed time (ms) for GetData - QUERY - query:a66c3df3-279e-4031-86fb-736e104d5ad8_c95 - [dd124922] = 464 Notice that the caching has really brought down the query times. We are definitely seeing benefits of caching. The very last entry for query: dd124922 is taking 209421 ms without caching, but is taking 464 ms with caching.   This query (dd124922) has taken 209421 ms (209 sec or 3.49 min) to run without cache. That is a long run time for a query and could be the reason why our VA report took 3:55 min to open without cache. Now, Let us drill deeper into the query itself.   Drilling into the hard hitting VA query We will need to get the BIRD xml from VA. Doing a ctrl+alt+B on the VA report will open a window as below:     Save the BIRD. If you study the BIRD, you will notice that each query is structured as a ParentDataDefinition block. A search on the query of interest: dd124922 will find us a block such as this:   <ParentDataDefinition childQueryRelationship="independent" status="executable" dataSource="ds54909" name="dd124922">   If you scroll a bit in the xml - you will notice one or more child definitions under the parent, in the form of a DataDefinition such as this:   <DataDefinition dataSource="ds54909" type="multidimensional" name="dd124923">   dd124922 is the parent query and dd124923 is the child query. VA runs the parent query. Viewer keys off of the ‘childQueryRelationship’ – if ‘independent’, it runs each of the child queries independently; otherwise, it runs the parent. A further search of the query dd124922 in the BIRD will tell us the visual name and label associated with the query.   <Graph graphType="bar" supplementalVisuals="ve124924" resultDefinitions="dd124912" name="ve124937" data="dd124922" label="Andamento Puntualità" action="ac195691" applyDynamicBrushes="yes" sourceInteractionVariables="bi124911 bi130428"> In this case the visual label is "Andamento Puntualità" and the name is "ve124937" Now let us go to the Query Diagnostic Tool in VA. Notice you will find a drop down with all the query info. Select the one associated with the above query which in this case is: c41: Andamento Puntualita - ve124937           You could actually run this query by clicking on the play button next to it. You can specify in the scrollbar below if you want to run it from cache or otherwise.         This is our hard hitting query and a careful study indicated to us that the query is associated with a big where clause which is where the maximum amount of time is spent during report open. Based on this piece of information, we reworked our backend data to optimize the combinations and the number of observations that was being passed to the report filter. This brought down the query time considerably both in VA and Viewer.      <queryRequests> <queryRequest data="dd124922" dataLevel="custom"> <expressions> <expression type="filter" dataStage="detail">or(and(eq(${bi54954}, 191)), and(eq(${bi54954}, 192)), and(eq(${bi54954}, 193)), and(eq(${bi54954}, 194)), and(eq(${bi54954}, 202)), and(eq(${bi54954}, 203)), and(eq(${bi54954}, 204)), and(eq(${bi54954}, 206)), and(eq(${bi54954}, 212)), and(eq(${bi54954}, 213)), and(eq(${bi54954}, 214)), and(eq(${bi54954}, 215)), and(eq(${bi54954}, 216)), and(eq(${bi54954}, 217)), and(eq(${bi54954}, 218)), and(eq(${bi54954}, 219)), and(eq(${bi54954}, 220)), and(eq(${bi54954}, 223)), and(eq(${bi54954}, 224)), and(eq(${bi54954}, 225)), and(eq(${bi54954}, 228)), and(eq(${bi54954}, 229)), and(eq(${bi54954}, 230)), and(eq(${bi54954}, 231)), and(eq(${bi54954}, 232)), and(eq(${bi54954}, 235)), and(eq(${bi54954}, 239)), and(eq(${bi54954}, 240)), and(eq(${bi54954}, 241)), and(eq(${bi54954}, 242)), and(eq(${bi54954}, 243)), and(eq(${bi54954}, 250)), and(eq(${bi54954}, 253)), and(eq(${bi54954}, 257)), and(eq(${bi54954}, 259)), and(eq(${bi54954}, 260)), and(eq(${bi54954}, 264)), and(eq(${bi54954}, 267)), and(eq(${bi54954}, 268)), and(eq(${bi54954}, 273)), and(eq(${bi54954}, 274)), and(eq(${bi54954}, 276)), and(eq(${bi54954}, 277)), and(eq(${bi54954}, 278)), and(eq(${bi54954}, 279)), and(eq(${bi54954}, 280)), and(eq(${bi54954}, 281)), and(eq(${bi54954}, 282)), and(eq(${bi54954}, 283)), and(eq(${bi54954}, 284)), and(eq(${bi54954}, 285)), and(eq(${bi54954}, 286)), and(eq(${bi54954}, 287)), and(eq(${bi54954}, 289)), and(eq(${bi54954}, 290)), and(eq(${bi54954}, 291)), and(eq(${bi54954}, 294)), and(eq(${bi54954}, 296)), and(eq(${bi54954}, 297)), and(eq(${bi54954}, 298)), and(eq(${bi54954}, 299)), and(eq(${bi54954}, 301)), and(eq(${bi54954}, 302)), and(eq(${bi54954}, 303)), and(eq(${bi54954}, 304)), and(eq(${bi54954}, 306)), and(eq(${bi54954}, 307)), and(eq(${bi54954}, 308)), and(eq(${bi54954}, 309)), and(eq(${bi54954}, 310)), and(eq(${bi54954}, 311)), and(eq(${bi54954}, 312)), and(eq(${bi54954}, 313)), and(eq(${bi54954}, 314)), and(eq(${bi54954}, 315)), and(eq(${bi54954}, 316)), and(eq(${bi54954}, 318)), and(eq(${bi54954}, 319)), and(eq(${bi54954}, 320)), and(eq(${bi54954}, 321)), and(eq(${bi54954}, 322)), and(eq(${bi54954}, 323)), and(eq(${bi54954}, 324)), and(eq(${bi54954}, 325)), and(eq(${bi54954}, 326)), and(eq(${bi54954}, 327)), and(eq(${bi54954}, 328)), and(eq(${bi54954}, 330)), and(eq(${bi54954}, 331)), and(eq(${bi54954}, 332)), and(eq(${bi54954}, 333)), and(eq(${bi54954}, 334)), and(eq(${bi54954}, 336)), and(eq(${bi54954}, 337)), and(eq(${bi54954}, 338)), and(eq(${bi54954}, 339)), and(eq(${bi54954}, 340)), and(eq(${bi54954}, 341)), and(eq(${bi54954}, 342)), and(eq(${bi54954}, 343)), and(eq(${bi54954}, 344)), and(eq(${bi54954}, 345)), and(eq(${bi54954}, 346)), and(eq(${bi54954}, 347)), and(eq(${bi54954}, 348)), and(eq(${bi54954}, 349)), and(eq(${bi54954}, 352)), and(eq(${bi54954}, 353)), and(eq(${bi54954}, 355)), and(eq(${bi54954}, 356)), and(eq(${bi54954}, 357)), and(eq(${bi54954}, 358)), and(eq(${bi54954}, 361)), and(eq(${bi54954}, 362)), and(eq(${bi54954}, 363)), and(eq(${bi54954}, 364)), and(eq(${bi54954}, 365)), and(eq(${bi54954}, 366)), and(eq(${bi54954}, 368)), and(eq(${bi54954}, 369)), and(eq(${bi54954}, 370)))</expression> <expression type="filter" dataStage="detail">and(eq(${bi54952}, 'LUN'))</expression> <expression type="filter" dataStage="detail">and(between(${bi118321}, 0, 23))</expression> <expression type="filter" dataStage="detail">and(eq(${bi54985}, 'O'))</expression> </expressions> </queryRequest>   Analysis of Viewer Queries   As is the case with VA, Viewer does not have a diagnostic tool to anlayze the queries in detail. However, we can use the following general guidelines while diagnosing Viewer queries:   Viewer query times should be similar to VA query timings. They should roughly be taking the same amount of time. The total number of VA queries for each page of a report should be similar to Viewer query numbers. For example: if VA total queries were 35 and Viewer was at 77: then this is an indication that something is not right and we need to do more investigation. We can get this information from the queries.txt file we created in the earlier step.  In Viewer, we should only be running the queries required for the first section when the report renders. Then, when switching sections, we will run the queries needed for the new section. If you are seeing queries being run for other sections, then that needs further investigation.    Conclusion Diagnosing performance-related issues is a complex process, requiring attention to detail. Applying this information will help you quickly diagnose issues when they arise.   ... View more

The data-driven content object enables you to display your data in a custom third-party visualization within your SAS Visual Analytics report. The third-party visualization can be authored in any JavaScript charting framework, such as D3.js, Google Charts, or CanvasJS. The visualization in a data-driven content object receives its data from SAS Visual Analytics and interacts with filters, ranks, and actions in the same way as other objects in your report. The data-driven content object lets you create custom, compelling third party visualizations that can represent data in unconventional ways, seamlessly integrating with SAS Visual Analytics.   There is a little bit of work required to accomplish this. In this article: we will build the step by step java script code to integrate a Google 3D-PieChart within your SAS Visual Analytics application.    Adding a DDC object in VA In  VA, create a new report. Add a data source and add a data-driven content object as shown below.                                  At this point no roles have been assigned. Notice that you see a greyed out table with some sample data in there. If you right click and select the “View Frame Source” it will give you the html for the default visualization for DDC objects. This (custom_table.html) visualization is provided to you by VA, and it uses the following URL to render the table: http://YourVIYAServer/SASVisualAnalytics/resources/custom_table.html The default custom data table is to be used as a placeholder while the DDC content object Roles are not set.             Now assign some roles and notice how the graph becomes active and interactive with the roles and data selections you make.  This  custom_table.html is a good place to start your learning, while you start writing custom java script code to integrate third-party visualizations within VA. Read thru the code to see what methods are called and how data is structured back and updated in VA.         Google 3D-PieChart Now let us see how we can render a google 3D-PieChart from within VA. The developers API reference page for google charts  is as follows: https://developers.google.com/chart/interactive/docs/gallery/piechart#making-a-3d-pie-chart This is my starting point. I will refer to  this for a lot of things such as the package name to load, data formats, configuration options, examples  etc. Keep this page handy and ready while you are writing code.   Notice the 3D-PieChart example on that page has code snippets. The code snippet is a simple standalone 3D-PieChart example which can render on its own. Simply, copy the code to a file and call it  google_3DPieChart.html and save it in the in the /var/www/html/ location of your VIYA midtier machine where all the VIYA microservices are running. It is the document root for the VIYA web server.    You could reference this html file as follows: http://YourVIYAServer/google_3DPieChart.html   For this example, I used the code for the 3D-PieChart example and made some modifications to the var names, added a few more rows to the sample data, added the path to the jquery-3.1.1.min.js and the jquery.csv.min.js libraries and added the “PieSliceText” label.   The modified code now looks like this:         In the VA report you created above, add a new page and pull down a Data Driven object to the canvas In the right hand side on the Options tab: add the link to your html file you created above.     You will get a greyed out 3D-PieChart with the sample data we have coded in the java script.  At this point if you go ahead and assign a variable to the roles tab, the graph will become active. However notice that it always displays the same sample data no matter what role and data selections you make. The data and the roles you assign do not get updated in VA. What you have is a stand-alone Web Content being displayed within VA. In order to add interactivity we have to use the windows Event Handler code, so that  VA can send and receive messages back from the third-party visualization.     event handlers If you want to catch user clicks, or need to manipulate properties or data in a chart that you've already drawn, you need to listen for events thrown by the chart. To achieve this, we need to have some mechanisms to detect what the user and the program are doing at any given moment—where the mouse is in the browser window, whether the user has clicked a mouse button or pressed a keyboard key, whether a page has fully loaded in the browser, and so on. All of these occurrences we refer to as events, and JavaScript has a variety of tools to help us work with them. JavaScript deals with events by using so-called event handlers The HTML5 API window.postMessage() method safely enables cross-origin communication between Window objects such as  between a page and an iframe embedded within it. event handlers catch user actions and make web pages interactive.   Add this piece of code to the google_3dPieChart.html file you created above.     Notice, the function onMessage calls a function called updateData(evt.data). This is a very important function and we will get to that in a minute.   The onMessage function is called each time the data for the data-driven content object is updated. The message received from the event handler comes as evt.data This is a JSON object and here are some of its attributes: resultName, data (the query results stored in a two-dimensional array), rowCount, columns and parameters.   For more explanation, refer to this blog post:  Data messages received from VA   Resizing Windows Next, we will need to add code to redraw charts when a window is resized. The graph should be able to adjust every time the user resizes. A window resize is an event caught by the windows event handler above.   jQuery has a resize event handler which you can attach to the window, .resize(). So, if you put $(window).resize(function(){/* YOUR CODE HERE */}) then your code will be run every time the window is resized.   You must also define the drawChart() function that will redraw the graph each time the window is resized. Add this code below for the drawChart() function.     updateData   This function processes the raw data received from VA and sends it back to google in a form that google understands. This is where you have to read the developer’s google chart documentation page to understand the format in which google would want the data back.  The code below is the simplest version of a simple two column table. You could add code here to catch error conditions, validate data rows and columns, send instructional messages etc. This is also the place where you will want to add customizations  for the different google graphs you will want to develop.   First: Add the following piece of code to set some global variables and chart options for the 3D-PieChart.     The updateData() function below is a simple 2 column table example. As I said for more complex data structures, you will need to read up on the google documentation and add code to manipulate raw data to a format google understands.         View it in VA Save the google_3DPieChart.html file with all the code additions. In VA do a reload of the iFrame or just reload the whole application and you should now see your 3D-PieChart come to life with the roles and data selections you have made. Here are a few screen shots of the finished visualization.        Same Visualization but with a different set of roles selection.            Exploded 3D-PieChart You can separate pie slices from the rest of the chart with the offset property of the slices option: To separate a slice, create a slices object and assign the appropriate slice number an offsetbetween 0 and 1. Below, we assign progressively larger offsets to 1, 2, 6 and 8 th slice (meaning rows of our data displayed as a slice in a PieChart). The offset of 0.2 indicates the distance of the slice  from the radius of the Pie.      Add this piece of code to your var "options" and var "_options" in the google 3D-PieChart.html example above.                   Same Exploded-3D-PieChart visualization but with a different sorting order.     Same Exploded-3D-PieChart visualization but this time with a different selection of roles.          Next Blog Post Well folks, this is all there is to get a simple google chart to display within VA. For many of the google chart examples, you could simply use the building blocks above, to leverage them from within VA. In cases where the data structure is more complex, code needs to be added to process the the raw JSON data coming back from VA.    If you want your third-party visualization to act as a source and a target to an interaction, we will need to handle Event Handler code to send selections, linked selections and some customized instructional messages. In my next blog post, I will be adding the SelectionHandler code to the example above: to create those interactions within VA. We will also be creating another slightly complex Google Chart example in my next article.    For added reading, please also refer to this article by Renato Luppi  on Data-Driven Content object.   Thanks! ... View more