Open-source data visualization: SASCTL, SWAT and Django

[#1]

In a previous article, I explained how you can use python programming language to build a dashboard where you can load data into CAS and visualize these data using the Dash python package. The article also covers how to score data using SAS Micro Analytics Services from python through the SWAT package.

In this article, I will cover the same concepts but this time using the Django framework. The main topics which will be covered are:

• How to access CAS data from a Django web application?
• How to score data from a Django web application and display the results?

What is Django?

If you are familiar with the python programming language, you may have already heard about Django. On the Django website, you can read the following statement:  The web framework for perfectionists with deadlines. To paraphrase the statement, Django helps developers to build web application by providing an opinionated approach to web development in python. How does that translate in facts? Django provides a lot of out-of-the-box functionalities which can be used by web developers to quickly start their project and reuse code and functionalities provided by the package. This might still seem obscure. Django provides an easy way to define:

• Views
• Models
• HTML pages
• Database integration

When you create a web application, you end-up writing and rewriting the same functionalities again and again. Django does all that wiring for you which gives the developers more time to focus on what is generating added value. To be more precise, the Views contain the business logic, what should be displayed when accessing a specific web url. The Models are an abstraction to database tables. HTML pages take the form of templates which can filled by data coming from the Models and manipulated by the Views based on the request send from the web browser. For the database integration, Django facilitates the integration by synchronizing the Models with the database of your choice (sqlite by default). In addition to these foundation elements, Django also provides an Admin interface which makes it easy to manager users, groups and Create/Read/Update/Delete (CRUD) operations on the database. Without any specific configuration, the Admin interface allows the users to browse the data and also to edit and delete data. As a developer, using Django helps you to focus on the code in your IDE and access the application in the browser. All the wiring between the web server and the database is handled by Django. This makes it a good choice for data scientists who are willing to expose their data on the web using their preferred language: python.

Setting-up the application

Setting up a Django project is simple. You should have python and pip installed on your machine. When done with python and pip installation, create a python virtual environment and activate it. You can then install Django using this command:

pip install django

After installation, you should execute the following command to create a project:

django-admin startproject mysite

In that command, you should replace mysite with the name of your project. In my case, I used django_mas as the project name. This command will create the structure of the project and generate most of the files needed to create the application. If you execute the following command, you will start the Django web server with a basic application running.

python manage.py runserver

As soon as the project is ready, you can add applications to it. Applications in Django can be considered as a set of related features for your web application. For example, if you are creating a web site for a library, you may create applications for:

• User management
• Book registration
• Book borrowing
• Membership management

The benefit of this approach is that multiple developers can work all together on the project and work independently on their application. To create an application, you should execute the following code:

python manage.py startapp myapp

Where myapp is the name of an application you want to create. As soon as you create an application, you should register it in the project. This can be done by adding the name of your application to the settings.py located under the project configuration folder (the folder inside your project with the same name as your project).

INSTALLED_APPS = [
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
    "myapp",
]

As you can see from the list above, Django already registered a few apps by default. These apps are needed by Django to work properly and implement a bunch of functionalities you will use without having to write code. At this point, you should have a better understanding about how-to setup a Django project and create an application. If you want more information about this process, the Django website provides a good tutorial to start building applications.

The application

You have now been introduced to Django. It's time to show you what can built using Django and SAS Viya. The application is "basic", but already implements a few techniques to integrate SAS Viya data into a Django based web application.

The dropdown boxes at the top of the page are retrieving data from a table stored into the CAS server. The two cards are populated based on the user choices. The card on the left displays data coming from the CAS server while the card on the right displays results from the scored data. To score the data, a model has been:

• created in SAS Visual Analytics
• registered in SAS Model Manager
• published to MAS

The application also implements another functionality which benefits from the integration of Django with a database. The web application registers information into a database to monitor the performance of the model. Storing the parameters passed to the model and result of the scoring, is important if you want to analyze the performance of your model across period of times. As you will see later, Django makes it really easy with just a few lines of code. Note: this demo application doesn't implement the user authentication. The username and password are hardcoded in .env file which is not included in the GitHub repository containing the code.

The code

As mentioned, the application uses two different sources of data: a CAS table and the MAS server. Let's focus first on the CAS related actions. To access the CAS server, you need to create a CAS session. As this will be used multiple times across the application, I've created a function for this under /django_mas/default/views.py:

def get_cas_session(request):
    if request.COOKIES.get("cas_session_id") != None:
        try:
            session_id = request.COOKIES.get("cas_session_id")
            cas_session = swat.CAS(
                f"{SERVER}/cas-shared-default-http",
                ssl_ca_list=CERTIFICATE,
                session=session_id,
            )
            return cas_session
        except Exception:
            print(f"Session {session_id} is not valid!")
    cas_session = swat.CAS(
        f"{SERVER}/cas-shared-default-http",
        username=USERNAME,
        password=PASSWORD,
        ssl_ca_list=CERTIFICATE,
    )
    return cas_session

As you can see in the code, the input parameter is a request object. This object is provided by Django and contains the information from the HTTP request. In our case, we are retrieving a cookie containing a cas_session_id. This session id is checked to validate if a session exists and is still active. If not, a new session is created. Reusing a session is a good practice to avoid starting too many sessions on the CAS server. As you can see in the code, we are using the SWAT package to create a CAS session. The SWAT package is provided by SAS and facilitates access to the CAS server from python. More information about the SWAT package and integration of CAS into python can be found here.

Creating views

The next function in the file is the one that will be to render the web page.

def index(request):
    cas_session = get_cas_session(request)
    cas_session_id = getattr(cas_session, "_session")
    cas_table = cas_session.CASTable(name="Cars", caslib="Public")
    template = loader.get_template("default/index.html")
    context = {"origins": cas_table.origin.unique()}
    response = HttpResponse(template.render(context, request))
    response.set_cookie("cas_session_id", cas_session_id, max_age=15 * 60)
    return response

This function creates a CAS session, accesses the Cars table located in the Public CAS library. An HTML template is then retrieved. If you are not familiar with HTML templates in general and with Django more specifically, you can find more information here. It will also be covered later in this article. We then define a context variable which uses the SWAT package behind the scenes. If you have already worked with Pandas dataframes, the syntax should be familiar to you. Finally, the function creates an HTTP response and sets the cookie with the session information. The next three functions defined in the views.py file are used to retrieve values based on the user choices. They are used to populate the different dropdown boxes and to retrieve the detailed cars information. As you will notice, these different functions are basically reusing the same structure and the one above. The biggest difference resides in the fact that instead of returning a HTML template, this time a JSON string is returned. This will allow the users to stay on the same page and to add interactivity to the web page.

def get_make(request):
    cas_session = get_cas_session(request)
    cas_session_id = getattr(cas_session, "_session")
    cas_table = cas_session.CASTable(name="Cars", caslib="Public")
    table_filter = f"origin=\"{request.GET.get('origin')}\""
    data = cas_table.query(table_filter).to_frame()
    json_response = {"values": data.Make.unique().tolist()}
    response = JsonResponse(json_response)
    response.set_cookie("cas_session_id", cas_session_id)
    return response


def get_model(request):
    cas_session = get_cas_session(request)
    cas_session_id = getattr(cas_session, "_session")
    cas_table = cas_session.CASTable(name="Cars", caslib="Public")
    table_filter = (
        f"origin=\"{request.GET.get('origin')}\"&make=\"{request.GET.get('make')}\""
    )
    data = cas_table.query(table_filter).to_frame()
    json_response = {"values": data.Model.unique().tolist()}
    response = JsonResponse(json_response)
    response.set_cookie("cas_session_id", cas_session_id)
    return response


def get_details(request):
    cas_session = get_cas_session(request)
    cas_session_id = getattr(cas_session, "_session")
    cas_table = cas_session.CASTable(name="Cars", caslib="Public")
    table_filter = f"origin=\"{request.GET.get('origin')}\"&make=\"{request.GET.get('make')}\"&model=\"{request.GET.get('model')}\""
    data = cas_table.query(table_filter).to_frame()
    json_response = {"columns": list(data.columns), "rows": list(data.loc[0])}
    response = JsonResponse(json_response)
    response.set_cookie("cas_session_id", cas_session_id)
    return response

At this point, you have defined all the functions/views needed to build the web page. You may now wonder how they can be accessed. Django defines the link between the view and a URL in a specific file named: urls.py (located in the application folder).

urlpatterns = [
    path("", views.index, name="index"),
    path("api/make", views.get_make, name="get_make"),
    path("api/model", views.get_model, name="get_model"),
    path("api/car-details", views.get_details, name="get_details"),
    path("api/price-prediction", views.score_data, name="score_data"),
]

The first url pattern defines the access to the root of your web site. In this case, http://localhost:3000/ The link is done with index function in the views file. And it will be identified as index. As you can deduct from the previous explanation, the second url pattern makes the link between http://localhost:3000/api/make and the get_make view. The last url pattern defines the link for the scoring view.

@csrf_exempt
def score_data(request):
    data = json.loads(request.body.decode("utf-8"))
    with Session(SERVER, USERNAME, PASSWORD, verify_ssl=CERTIFICATE):
        result = mas.execute_module_step("msrp_prediction", "score", **data)
        record = ScoredData()
        predicted_data = {**data, **result}
        for key, value in predicted_data.items():
            setattr(record, key, value)
        record.save()
        response = JsonResponse(result)
        return response

The code gets data from the request object. It then creates a session on MAS. To achieve this, we use the SASCTL package. This is a python package designed to ease the model management related tasks from python. For more information about the SASCTL package, please refer to the repository. A request is then made to score the data and a response is then returned to the browser. In the view, we are also saving the data to a database. This is achieved by the ScoredData model that is used to create a record and then save it to the database.

Creating the model

Django provides the infrastructure to handle data models and make the integration with a database transparent. In our case, we have created a model in the models.py file.

class ScoredData(models.Model):
    ID = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    Type = models.CharField(max_length=30)
    DriveTrain = models.CharField(max_length=30)
    EngineSize = models.FloatField()
    Cylinders = models.IntegerField()
    Horsepower = models.IntegerField()
    MPG_City = models.FloatField()
    Weight = models.FloatField()
    Wheelbase = models.IntegerField()
    P_MSRP = models.FloatField()
    Created_At = models.DateTimeField(default=timezone.now, editable=False)

The model will be responsible to store the data in the database defined in the project settings.py. In the default configuration, it will point to a sqlite database, but you can configure other databases based on your needs.

After updating the models.py file, you need to synchronize the database. This is done by executing the following commands from the root folder of the project.

python manage.py makemigrations

python manage.py migrate

As soon as you execute these commands, you will be able to create new rows in the database using the score_data view. The next step is to create the HTML page which will be rendered by the index view.

Using templates

To build web pages, Django uses a templating engine. This means you can create the structure of the HTML page and pass data to it. The templating engine allows also to split the HTML page into multiple blocks which can then be rendered conditionally. Following the logic above, the index view renders the index.html  which is defined as a block:

{% block content %}

...

{% endblock %}

The block contains HTML code to render the static elements. It also includes the script tags which contains the JavaScript used to create the interactivity on the page and avoid page reloads. This will turn the application into a Single Page Application (SPA). The JavaScript implements the different functions required by the interface. The code is documented in the repository if you want more information about it. At the top of the index.html file, we indicate that the file extends the content of layout.html.

{% extends 'default/layout.html' %}

The layout.html  integrates Bootstrap library in the application and defines a navigation bar. Using Bootstrap assures consistent styling of the application in the web page and eases the development thanks to predefined CSS classes. For more information about Bootstrap. At this point, you have everything you need to run the application as demonstrated earlier.

Using Django Admin

Now that you have been testing your application, you might be interested in checking what was inserted into the database. You can therefore use the Django Admin page. To enable the Django Admin functionality, execute the following command to create a super user.

python manage.py createsuperuser

This command will prompt you for some information about the user you want to create. Don't forget the credentials as it will be your administrative user. You should then define the admin url in the urls.py located in the project folder:

urlpatterns = [
    path("", include("default.urls")),
    path("admin/", admin.site.urls),
]

To adapt the display in the Admin page, you should define the list of variables you want to display.

class ScoreDataAdmin(admin.ModelAdmin):
    list_display = (
        "Type",
        "DriveTrain",
        "EngineSize",
        "Cylinders",
        "Horsepower",
        "MPG_City",
        "Weight",
        "Wheelbase",
        "P_MSRP",
        "Created_At",
    )

admin.site.register(ScoredData, ScoreDataAdmin)

You are now ready to view the recorded data in the admin page.

Conclusion

While it is easy to create a web application using python and Django, you need to have a bit of JavaScript knowledge to build quality looking interfaces especially adding user interactivity. Using python for web development, interaction with CAS, scoring with MAS, is made easy thanks to python packages: SWAT, SASCTL and Django. You can build any custom application you want using these. Of course, the demo application doesn’t make use of authentication, but it can be easily added thanks to Django framework. For more information about building web applications using Python or JavaScript only, please refer to the following articles:

Open-source data visualization : Dash and SWAT

Open-source data visualization : Dash, SWAT and SASCTL

An approach to SAS Portal in Viya

My first development with VA SDK using Node.js

Using Node.js to develop with the VA SDK – a deeper dive

Deeper integration of SAS Viya SDK content into your custom web application

Find more articles from SAS Global Enablement and Learning here.