Game on! Cortex: An Analytics Simulation Game

3 Likes

The purpose of this blog is to introduce the Cortex Analytics Simulation game. The game is available as a learning tool to SAS’s commercial and academic customers and partners.

The Purpose of Cortex:

The purpose of the Cortex game is to teach predictive modeling and machine learning concepts in a hands-on, fun, yet competitive environment. Cortex breaks from the traditional instructor led training class which typically consists of lecture followed by computer demonstration. That’s why it is referred to as a game and not training. In the Cortex game, players are thrown into an analytics sandbox and given a bunch of predictive modeling toys to play with, at their own pace. Through a Cortex game page, players have access to a real-world business case study as well as online instructions and tutorials, in the forms of documents and videos. They can also create their own virtual lab reservations where they have access to SAS software and all data needed to play the game. Further, during the duration of the game (typically about 2 weeks, but more on that later), players have access to live instructors/mentors through pre-scheduled office hours. Finally, players have access to an online leaderboard where they submit results and get immediate feedback to see where they stand against other players in the game.

One huge benefit of Cortex is that no modeling experience is necessary. Read more about this in the “Playing Cortex” section below. Further, each organization plays their own private game with their own unique leaderboard so there is as much team building happening as there is learning! Below is a screen capture of the game page for one of the Cortex scenarios.

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

The History of Cortex:

Cortex was first developed at the university level by the Canadian business university HEC Montreal, in partnership with SAS. A small group of business professors came up with an idea for an out-of-the-box way for analytics and data science students to learn about building predictive models without using the framework of a traditional classroom. They wanted their students to jump into real-world data with minimal guidance and build models using trial-and-error. Simply put, they wanted the students to learn by doing. To make it fun for college students the idea was to treat it like a competition with a leaderboard to display winning entries. The students/players submit lists of customers the hypothetical company should contact and calculate an over-all profit or operating surplus to rank players. This learning method worked so well in the academic space, that SAS decided to make the game available for its commercial customers. It has been a huge success.

The Scenarios of Cortex:

The first application developed for the game used SAS Enterprise Miner. The scenario in this first application is a non-profit organization that is seeking donation contributions. Players build models to predict which potential donors to contact during a targeted marketing campaign. This fundraising scenario consists of two rounds. In the first round, players predict donation amount and market to those with the largest predicted value for amount. Operating surplus from contacting these potential high value donors is calculated on the leaderboard as players submit lists of possible donors to contact. More advanced topics are used in round two of the game where the players consider two targets: a binary target of whether customers donate or not, and a numeric target of how much they donate if they do. Such an application is known as two-stage modeling. In addition, the concept of Net Lift (also called Uplift or Incremental Response) modeling is introduced. The idea in Net Lift modeling is to investigate which potential donors can be persuaded to donate by being contacted, and then market to these persuadable donors. Does being contacted provide a lift, or increase, in the probability they donate or how much they donate?

Over time, HEC Montreal found such success with Cortex, that they decided to expand the game to other scenarios and analytical applications. With help and additional development from SAS, the games were extended into SAS Viya. Below is a summary of all current scenarios:

• Fundraising: The Fundraising scenario is described above. As it was the first scenario developed, it is the most mature of all the applications. Versions of this scenario exist for SAS Enterprise Miner, Model Studio on SAS Viya, and also a coding and open-source version which uses applications like SAS Studio and Jupyter notebook.

• Customer Retention (Available for SAS Model Studio and SAS Enterprise Miner): Players are working for a fictitious tele-com company that is trying to prevent customers from leaving, also known as customer churn. Predict which customers have a high chance of churn and try to get them to stay! Part of the scenario is to investigate whether inviting high risk, current customers to a high-end dinner party will prevent them from churning.

• Credit Risk (Available for SAS Model Studio and SAS Enterprise Miner): Players work for a financial services company that is deciding to which customers credit will be granted. Players of the game predict customers’ creditworthiness based on demographics, financial indicators, historical loan information and credit related metrics. Players will use the Credit Scoring add-ons for Model Studio or Enterprise Miner to predict which customers should be approved or rejected for the loan. Part of the scenario allows the players to apply the concept of Reject Inference to the game. Two powerful tools available in the Credit Scoring add-ons are used in this scenario: the Interactive Grouping node and the Scorecard node. These nodes in the prebuilt pipeline for the Model Studio version of the game are shown below.

A typical timeline for a Cortex game:

Academic applications of the game can be very customized in terms of how long they run. An instructor could choose to run the game over several weeks or months, where students play the game periodically while receiving traditional classroom-based instruction. Or they may decide to have students fully immersed in the game for full-days of gaming activity, where the entire game would be run in just one or two days. The timeline for commercial games is usually a bit more consistent.

Commercial games generally take just over two weeks to run; about 2 weeks of actual game time, plus an additional kick-off session at the start and a wrap-up session at the end. There is an initial 1-hour kick-off session typically in the week prior to the game actually being played. In the kick-off, students are introduced to the game scenario, given instructions on accessing all game needs (the game page, how to create virtual lab reservations, the leaderboard) and a few pointers on how to get started. During the following two weeks, players play the game when and for how long they choose. The more time players put into the game, the more they learn from it. This is one of the big benefits of being outside the traditional classroom. Players make their own lab reservations, and they play the game when convenient for them.

During those two weeks of playing the game, there are four, 1-hour, instructor led office hours. No two office hours are ever alike. Instructors typically start by discussing the leaderboard and the standings for the players. Then the instructors address any and all questions the players have. Finally, time permitting, the instructors can give pointers on predictive modeling concepts or specific details about using the software, all with the intent of helping players be more competitive in the game. The leaderboard shuts down at the end of that two-week period. The game then concludes with a 1-hour wrap-up session where the winners are congratulated and, in some cases, awarded prizes. All players are given the opportunity to discuss their experiences playing the game, what they learned, and what worked and what didn’t work for them to be successful.

Playing Cortex:

Getting started playing the game is easy. For each game, players are given a pre-built project or pre-written code as a starting point. So even players brand new to predictive modeling or that have never used advanced software before can sit down, basically “click run”, and get started. They’ll have two predictive models trained, assessed, and compared immediately upon starting the game. Below is an example of the out-of-the-box pipeline available in the pre-built project for the Fundraising scenario.

Cortex takes a crawl, walk, run approach as the game progresses. All scenarios for the game have two distinct rounds. Round one for each scenario typically covers the basics: predictive modeling and machine learning fundamentals and getting players comfortable with new software. Round two typically covers more advanced analytical methods, beyond those which are covered in round one. Maximizing earned profit, in other words, being competitive in the game, typically is done by applying the round two concepts.

The Leaderboard:

We have a winner! A major component to the game is the online leaderboard, which is unique for each game. Players submit a list of customers to the leaderboard. This mimics a real-world situation in which a data scientist submits a list of customers to contact in a marketing campaign or to grant credit. A data set with known answers (for example in the fundraising scenario, who donated and how much money) is held back from players. This hidden data set is used to calculate operating surplus or over-all profit based on the specific list of customers that the players submit. Players can submit up to 100 different times to try to improve their score and get the coveted #1 spot on the leaderboard. In the screen shot below, from a hypothetic game, 9 players have made submissions, where expenses and operating surplus are calculated for each submission. The player with highest over-all operating surplus is in first place.

To sign up for the game:

If you are a SAS customer and would like to play Cortex with your team, fill out this form and we’ll be in touch soon: https://www.sas.com/en_us/training/forms/cortex-for-business-pricing-form.html

If you are at an academic institution and you are interested in Cortex for your class, send an email here: Cortex@hec.ca