Character customer attributes with a large number of distinct values are often the culprit when validation takes a very long time.   In the Input Data Wizard, you can select the customer attributes that you want to load into SAS Marketing Optimization. But we are finding that many people (myself included) tend to use the “Select All” since it’s easier than individually picking each attribute that you want to load.   If all of your attributes are numeric, or if the character attributes really are categorical attributes (like Gender or State, with not too many distinct values), then selecting more attributes than you really need shouldn’t affect the performance too much.   But if one of the attributes is something like address or social security number, with millions of distinct values, then the validation will take a *really* long time because all of the individual values need to be validated and loaded into the MO datastore.   Use numeric data or categorical attributes Refresh will take even longer because even deleting the data from the datastore takes a while, and refresh actually does both (i.e., creates new data and then deletes the original). But since you would never really want to use any of these values in a filter (in other words, you would never create a constraint that applies only to customers where address=”123 Main St, Cary, NC”), you’ll get much better performance if you’re very careful when selecting customer attributes to only select those that are either numeric data or categorical attributes.   Delete old scenarios If the initial validation is fast but the refresh process is slow, it probably means that you have a lot of scenarios using the input data. When the refresh runs, each of the scenarios must be recreated against the new data, and validation is performed at the scenario level to make sure that the scenario elements are still valid with the new data.   Delete old scenarios that you no longer need to speed up the refresh process.       ... View more

  This document describes the concept of horizons for an MO-MA integrated environment via a simple example.   Assumptions: The reader is familiar with creating and using MOMA data&colon; Campaigns and campaign groups Optimization groups Preferred scenarios Scheduled executions   The concept of horizons came about when customers wanted a way to efficiently use computer resources when running a MOMA optimization.  Some customers have lots of campaigns in a campaign group which results in the processing for a huge amount of data.  The problem is that not all of the campaigns may be scheduled for execution during an optimization.  So we are processing a lot of extra information that will not be used for anything.   Using a horizon in your MOMA data ensures that optimization runs only for campaigns in the current horizon.  If a horizon period is defined, then only those campaigns that are scheduled to execute during the horizon will be loaded into the MO tables.  Therefore, MOMA will only be processing relevant information for each optimization.   When an optimization run starts, the horizon is used to determine which campaigns will be included and which campaigns will not be included.  This will result in campaigns being added to the MO input tables as well as campaigns being removed from the MO input tables.    Setting an optimization schedule If you plan to use horizons, the campaign group must have a schedule in place.  In MA, when the campaign group is created, it must have a schedule set at the group level, which is the schedule for the optimization. This means that when the scheduled time for the group arrives, the following happens:   MA recreates MO tables based on the latest information (including the current horizon) MO refreshes the optimization group using the new tables provided by MA Preferred scenario is optimized and promoted MA gets the promoted solution and uses the results for the optimized counts   When setting the optimization schedule for the group, schedules can also be set for the member campaigns.  The schedule for the campaign indicates the time (and recurrence) for the campaign to be exported.  When the scheduled time for the campaign arrives, the following happens:   Campaign is published and exported An export file is created using the optimized counts from the MO promoted solution   There are a number of rules to keep in mind when scheduling campaigns and campaign groups: A campaign can only be scheduled to run after the first optimization has occurred A campaign can only be scheduled to run once after each optimization   Setting the horizon Although the horizon is set in MO when creating the optimization group, determining the horizon number of days requires some collaboration between the MA user who is creating the campaign group and the MO user who is creating the optimization group.  Typically, this is the same person, but this may not always be the case.    Once a schedule has been determined for a campaign group and member campaigns and all other options have been assigned, the campaign group is saved and made available for MO.  When the optimization group is created in MO, there will be an option to enable and assign the horizon:     Selecting the number of days in the horizon can be tricky.  A few things to keep in mind are:   Using a horizon number of days that is too small can result in no campaigns in the horizon. In this case, then the optimization group is created or refreshed, there will be errors (conveyed by error messages in the UI and/or the log) that there are no campaigns in the data   Using a horizon number of days that is too big means that the current data may not need to utilize the horizon feature. If the horizon number of days is so large that all campaigns are always included in all optimizations, then that is an indication that horizons do not need to be enabled at all.  If this is the case, horizons can be disabled.  For disabled horizons, all campaigns are included in all optimizations.   Once the optimization group has been created with the horizon number of days, the number of days is sent back to MA, but is not surfaced anywhere.  MA does know that the campaign group is using horizons and the number of days, but this information is not available via the MA UI.  The user will have to check the value via the MO UI.   Example of MOMA process using horizons We have a campaign group with 3 campaigns which have an optimization group that is set up as follows:   Horizon number of days = 14 days Group scheduled every 4 days, start 4/1 Campaign1 scheduled 5 days, start 4/1 Campaign2 scheduled weekly, start 4/7 Campaign3 scheduled every month on the 15 th , start 4/15   The schedule for this would be as follows, where the shaded days represent some action (optimization or execution) on that day:     The daily activities can be broken down into more detail.  Dates in red indicate dates where a campaign has been added to the horizon or where a campaign has fallen out of the horizon.     Ramifications of horizons Because campaigns will be falling in and out of the horizons, there may be some unexpected behaviors and results with campaign group and scenario set up.  As of 5.4, dealing with these issues are left up to the user.  This means that the user should be familiar with the data being used, as well as how the horizon affects the data.   Some issues that the user may come across include the following:   Infeasibility of the preferred scenario due to constraints When using horizons, it is possible to create constraints that cannot be fulfilled for a particular horizon. For example, say we have Camp1, which has 50,000 eligible offers and Camp2, which has 100,000 eligible offers. These are put into a campaign group, where a group level constraint is created such that the minimum number of offers for the group is 100,000. Then say that, while using horizons, we have an instance where Camp1 is the only campaign in the optimization group.  This means that there are only 50,000 eligible offers.  However, we still have the constraint on the group where the minimum number of offers is 100,000. Since we cannot fulfill this constraint, the scenario will come back as infeasible. Best practice in this case is to make sure that the user does not create constraints like this which may cause infeasibility.  Group constraints may not be the ideal constraints to use when horizons are in play.  The same is true for channel constraints, which are, in essence, a type of group constraint.   Infeasibility due to filters in constraints or contact policies In MO, the user can create constraints and contact policies that use filters. For example, a scenario may have the following contact policy: min contact >= 1, where communication_cd IN [EmailMsg] Using our previous example, let’s say that the communication EmailMsg is a communication that exists only in Camp2. While using horizons, we have an instance where Camp1 is the only campaign in the optimization group.  So Camp2 is not in the optimization group.  We will still see the values for items in Camp2 as selections.  For example, EmailMsg will still be a valid option.  But since Camp2 is not in the optimization group, there are no eligible offers where communication_cd is EmailMsg. For this example, this means that the preferred scenario will come back as infeasible.  The contact policy we had will never be met since we do not have any offers where communication_cd is EmailMsg. This is also true of filters that are custom details.   Unexpected behavior and results due to filters in constraints or contact policies In a similar vein to the previous issue, you may get unexpected behaviors and results with filters that are falling in and out of the horizon. Using the same example, let’s say we have a constraint with a filter: max contacts <= 2500, where communication_cd IN [EmailMsg] As before, the communication EmailMsg exists only in Camp2. While using horizons, we have an instance where Camp1 is the only campaign in the optimization group. So Camp2 is not in the optimization group.  We will still see the values for items in Camp2 as selections.  For example, EmailMsg will still be a valid option.  But since Camp2 is not in the optimization group, there are no eligible offers where communication_cd is EmailMsg. For this example, the preferred scenario will still be optimized successfully.  But the user should be aware that the value of this constraint will be 0.  This will still satisfy the constraint, but it may not be what the user was expecting. This is also true of filters that are custom details.     ... View more

MATables are SAS datasets used by Customer Intelligence to represent the population (customers, households, etc.) of a node in a diagram. For example, if the count on a select node selecting customers is 3,456 then it will have an associated MATable containing 3,456 rows, each row containing one customer ID. These MATables are retained after the count is completed so that later executions of downstream nodes won't have to recreate the node population. This helps performance but can lead to exports being created using out of date information if export nodes and communication execution are used unwisely.   When are MATables created and deleted? Each time you execute a node in a campaign, whether by scheduled execution, interactive execution or using the nodes' "run" or "update counts" action, a new MATable is created for that node. If the node had a count prior to this execution, then an MATable was associated with that node and is replaced by the new MATable. MATables that are no longer associated with a node are deleted when the campaign is closed. If you saved the campaign prior to closing it, then the old MATables are deleted. But if you do not save the campaign prior to closing it, the new MATables will be deleted (since the campaign is reverting to its prior counts/node populations). MATables associated with nodes in a campaign are also deleted when the campaign is deleted. While there are some exceptions, in general a saved/closed campaign will have one MATable associated with it for each node in the campaign that has a count on the node.   How can I reduce the number of MATables being saved? Clear counts on campaigns before saving them unless those counts/populations will still be correct when the campaign is next used (i.e. the underlying data will not have changed). Delete old campaigns that are no longer being used (or at least open them, clear counts and save them if an archival copy of the campaign is needed). When are new tables created to replace old ones? It is important to remember that an MATable represents a node population at a point in time. If an existing MATable that was created a month ago is used instead of replaced by a new one, then the population as identified a month ago is used (i.e. does not reflect changes in the underlying data since the last execution.) MATables are created only when a node is explicitly run or when a node does not already have a count/MATable and a node below it in the diagram is explicitly run. Using the 'run' or 'update counts' feature on a node or nodes creates new MATables only for the node(s) selected. SAS Marketing Automation supports several use scenarios so you have several options to reuse MATables versus replacing them with new versions: One usage scenario is designing a new campaign. You often update counts on a newly-added node to see the size of the population at that point in their diagram. In this scenario, you probably don't want to rerun all nodes upstream from the node just to get an idea of the population of the newly added node. Another scenario is building up a campaign interactively to produce a set of exports to be used immediately. For example, you build quick campaigns to produce timely emails to customers after some event, building the campaign as the event progresses, fine-tuning each node to achieve the desired size of contact group, and then you want to execute a communication to that group without having to rerun the nodes on which they have already created counts/MATables. This use scenario is achievable by executing a communication with the "use most recent data when executing communications" options deselected. If executing communications, either via the scheduler or interactively, you may instead wish to recreate node populations at execution time to ensure that all changes to the underlying data since the last execution are reflected in the exports produced. This is accomplished by executing the communication with the "use most recent data when executing communications" option selected in the campaign, so that all old MATables are discarded and replaced by new ones. This ensures that the execution reflects the most current state of your DBMS. Execution of a campaign, scheduled or via 'Execute now' in the execution dialog, clears counts on all nodes in the campaign prior to executing the communications since it is assumed that executing the entire campaign should always reflect the most recent state of your database. Re-execution of nodes in a linked campaign is governed by the designer of the linked campaign via selection/deselection of the "use most recent data when referenced by a link node" checkbox in the linked campaign. Saving the linked (target) campaign without counts also ensures that execution of campaigns referencing the target cell/campaign via a link node will trigger creation of current population MATables (since linked campaigns are opened read-only and any MATables produced during their execution will be discarded when the linked campaign is closed.)   Can I just delete all MATables older than X months? In early MA releases, deleting MATables would cause "missing table" failures during execution if the deleted table was referenced. This occurred because checking for the existence of an MATable required executing a stored process, which greatly slowed opening campaigns.   In later releases (starting ~5.4), we decided that it was worth the performance hit on opening campaigns to avoid the execution failures and the existence of all MATables is checked when opening a campaign. You should now just receive a warning when opening the campaign; the node with which the deleted MATable was associated and all nodes following it in the diagram will lose their counts. Deleting or clearing counts on old campaigns is a safer option, though. ... View more

Another SAS Communities Library article addresses MATables -- this one: What are MATables in SAS Marketing Automation and how to I manage them? Well, here is a bit more information on why we use MATables to help improve performance.   SAS Customer Intelligence (SAS CIS) maintains holding tables to support the execution of campaigns. These holding tables are SAS data sets held in directories on a file system available to SAS CIS and are known as MATables from the name of the libref that specifies the location of the holding tables.   The MATables library is defined in SAS Management Console Data Library Manager, and best practice is to assign a separate MATables library to each business context from the Data Options settings in the Business Context properties dialog box in SAS CIS.   When executing campaigns, the flow of the campaign may dictate the use of MATables datasets in queries against the tables on the database. By default when SAS joins datasets with database tables, all data is returned to SAS to enable the join. This can take a great deal of time. To optimize performance when it executes campaigns, SAS CIS uploads these MATables temporarily to the customer database in order to support in-database joins. The system builds LIBNAME statements as, and when, they are required to enable the uploading of the MATables to the database. These dynamic libraries are named and referred to as DBTMPLIB.   When a join between an MATables dataset and database tables is required, SAS CIS will create a DBTMPLIB library then load the dataset onto a temporary table on the database (using the database’s bulk loading functionality if configured), execute the SQL, drop the temporary table and then clear the DBTMPLIB library. This passes the processing to the database for optimal performance and avoids the transfer of database data to SAS to enable the join.     ... View more

You tweaked something in SAS Marketing Optimization that seemed very trivial and now your objective value or eligible offers are suddenly different. What happened?   SAS Marketing Optimization is not an exact solver. In fact, no exact solver exists today that can solve problems on the scale of typical MO problems – often more than hundreds of millions of binary decision variables.   Rather, MO is a heuristic approach. This means that there are many ways to produce slightly different results for the exact same scenario. For example, MO is an iterative algorithm that uses stopping criteria to determine when to terminate iterations. Therefore, if you use the mo_opt_config.sas file to change the stopping tolerance or the number of iterations, the solution will be different. Or even if you change the order of the data in the customer table, the solution may be different. This is all part of the nature of a heuristic approach or any algorithm that uses approximations.   Yes, it is true that with an exact solver, you would not see a lower objective after removing two constraints. However, please keep in mind that with a heuristic approach, any number of changes to the problem or the data can result in a different path through the algorithm, and therefore a different solution. In this case, when you remove two constraints that aren’t binding, it’s easy to say that the objective should be better because the constraints are not binding. But that’s because you know after the fact that the constraints are not binding.   At the beginning of the optimization process, MO does not yet know that the constraints will not be binding, so it does need to use the constraint information in the early iterations – until it has enough information to conclude that the constraints are not important. So it takes a slightly different path through the algorithm than it does in the case where the constraints are not present. And because the algorithm is a heuristic approach, a slightly different path can result in a slightly different solution – meaning slightly better or slightly worse. A difference of 0.18% is well within the normal tolerance of what we would expect from a heuristic approach.   For most MO problems, there is no way to know the exact optimal solution (except perhaps in cases where the constraints are very simple and the optimal solution could be obtained by sorting and using a DATA step). Therefore, there is no way to quantify MO’s “inexactitude.” This is why we always try to encourage users to make comparisons based on the best possible alternative without MO rather than trying to compare against some theoretical optimum that may not be achievable.   For example, if using MO produces an X% increase over your best alternative solution (including possibly other marketing optimization software), then you should not be concerned about whether an exact solver would have been able to produce a Y% increase, unless you have access to an exact solver that can actually do so.   To recap: SAS Marketing Optimization is not an exact solver, and there are good reasons for that. What’s the most interesting thing this software enabled you to do? Share in the comments.   ... View more

If you regularly use SAS Customer Intelligence Studio (SAS CIS), you will see that some of the data items in your information map can be selected from a drop-down list and others can’t. For example, Gender is a good example of when you see a list and Address is one for which you never see a list.   This list is what we call Customer Intelligence Metadata (not to be confused with Metadata Servers or SAS Metadata Repository.)   SAS CIS has this metadata functionality for two reasons:   1 – To make it quicker and more accurate for you to select fields. Wouldn’t you agree it is easier to select the word FEMALE from a drop down list than to have to type it in all the time?   2 – To help you approximate how many items fit in certain groups, which may help you make better selections when designing your campaigns.   For example, when you have metadata set on Gender, you may see something like the screenshot below, an approximate count and percentage of how many subjects (Customers) fall into each category. This can help you verify that the counts you are expecting are valid.     Metadata Generation is the SAS process that updates the tables that store this CI Metadata. You decide how often you run this process, which is purely for usability. When the campaign/nodes are run, the actual database is accessed to get the counts, so there’s no need to update every time the database is refreshed.   Metadata exists to give the campaign builder an estimate of the data behind the scenes. While it doesn’t have to be 100% accurate, it can be if the data is refreshed daily and you generate metadata daily. Is that necessary? If the system can go a week and still be usable from a campaign-building perspective, then don’t create extra work for yourself. Just update it weekly.   Bottom line: Don’t worry the next time you run a count on the number of females you have and the count you get in the node differs from the count in the properties of the node. This is normal. It probably just means your data has been refreshed and the metadata generation process hasn’t yet been updated!   Top Tip: When you think about fields on which you want to generate metadata, only use data items with a small number of unique values.   Examples of fields that are great for metadata include:   STATE – as in Texas, Florida etc TITLE – Mr, Mrs, etc STATUS – Married, Single etc   Never use these fields:   Zip code Address Email address Telephone number ... View more

When you log a track with SAS Technical Support, one of the things we might ask for is a ViewRegistry Report. These reports tell us exactly what you have installed on your system and help us determine what hotfix releases you have. They are easy to produce and would save us both time whenever you need help with SAS Customer Intelligence.   It is important we get them from each of your servers. For example, it needs to be run on the compute server, web application server and the metadata server!  So, yes, we expect to see one hotfix report for each server you have!   Read Usage Note 35968 to find out how to run a ViewRegistry Report. ... View more