SAS Data Mining and Machine Learning topics

In Pipeline comparision tab, what is the use of holdout data set

kbharat439 — Mon, 10 Aug 2020 17:55:20 GMT

Please explain the use of score a holdout data set.

Solution for imbalanced data, categorical target, 98% weights on 0 outcomes.

akmsharif — Mon, 10 Aug 2020 06:14:36 GMT

I am trying to do logistic regression, decision tree, KNN & neural network on a dataset where I have 9800 rows, the target is binary and 98% 0. I have 1000 interval predictors, all the variables have many 0s and not normal distributions. How should I approach to handle the imbalanced data in SAS Miner for each of the models? Can somebody pls help?

Change the champion model

kbharat439 — Thu, 06 Aug 2020 11:22:47 GMT

Please let me know the process of changing a champion model in the pipeline comparision tab.

SAS Softwares

Sha8 — Wed, 05 Aug 2020 01:42:44 GMT

Hi,

I want to know the SAS softwares that could perform coding and performing a sort of thematic

analysis to identify themes. I know some SAS softwares that identify themes on the basis of number of counts and frequencies of a variable in the big data but I need a sort of coding analysis like Nvivo does and I do not need analysis on the basis of number of counts of a variable.

I want to collect information from the different online platforms that includes social networking sites, customer reviews and companies website. Can anyone identify few SAS softwares that pull the data from different social networking sites and companies website and also SAS softwares for identifying themes like Nvivo does and as well as SAS softwares for the analysis purpose? I also want to conduct semi structured interviews and I want to identify patterns through coding therefore I would be needing SAS software to perform coding and identify the themes. That would be good if anyone could mention the possible list of softwares that meet my research needs as mentioned above. I would really appreciate your assistance.

Thank you,

Kind Regards,

Shabana

Binning and Pre-Binning in Interactive Grouping

ronya — Tue, 04 Aug 2020 12:03:58 GMT

Hello all, I'm exploring the use of interactive grouping in SaS EMiner as a method to bin the values of interval characteristic and wish to ask about the pre-binning process. Why do we need to use the quantile or bucket method to pre-bin the interval variable values rather than apply Tree-based binning to the interval values directly?

Free Webinar: How Do I Integrate SAS® Viya® and Open Source?

MelodieRush — Mon, 03 Aug 2020 17:32:08 GMT

Hi Data Mining Community,

I’m presenting a live “Ask the Expert” webinar on August 18, 11 AM – Noon ET. SAS delivers an open analytics platform, built on the latest cloud technology and accessible from the interface or in the coding language of your choice, giving you the freedom to experiment and create. Combining the power of SAS with open source technologies, you can unify disparate tools and analytic assets into a streamlined, collaborative environment – fostering productivity, business agility and tangible results. Join me as I discuss the many ways SAS Viya integrates with open source.

You will learn how to:

Access the power of SAS using your existing skills, like SAS, open source or other programming skills.
Use Python or R in the analytical flow of pipelines.
Use Python or R through the SWAT package.

Click here to register for the webinar.

Want more tips? Be sure to subscribe to the Ask the Expert board to receive follow up Q&A, slides and recordings from this and other SAS Ask the Expert webinars.

Can't join the live event? You can view this and other Ask the Expert sessions on-demand here.

Want more tips? Be sure to subscribe to the Ask the Expert board to receive follow up Q&A, slides and recordings from this and other SAS Ask the Expert webinars.

Can't join the live event? You can view this and other Ask the Expert sessions on-demand here.

How to use an existing score code in a new pipeline (build models viya)

acordes — Wed, 29 Jul 2020 15:00:01 GMT

I want to derive a scoring code (i.e. from a cluster that I'd learned before) and assign this new variable an input role or filter role.

So I try to insert a sas code node and run the proc astore that works fine in SAS Studio.

But how can I achieve that the new variable shows up in the training data set?

/* SAS code */
proc astore;
score data=&dm_data
rstore=MODELS._654S34YEIM3X7NLUBHJ8PQRK2_AST
out=&dm_data_outmodel; 
run;

Setting variable weight on decision tree using SAS Miner

Livia_L — Tue, 14 Jul 2020 20:14:12 GMT

I have a data set with a binary target where only 1.3% of the members are marked as 1, while 98.7% are marked as 0, and I need to run a decision tree on that set. So, I tried two ways of doing that:

1) I created a simple random sample on SAS with 30% of the members marked as 1 and 70% marked as 0, and also created a "weight" variable on the data set, where the weight of the members marked as 1 was 1 and the weight of the others was 22 or something like that. Then, when defining my data source on SAS Miner, I assigned the frequency role to my weight variable. I tried to run the decision tree on that set, but it only returned one node

2) I read the Decisions Node Example from this page https://documentation.sas.com/?docsetId=emref&docsetTarget=p0ak10c0ywr8mrn1an7tum03sp7g.htm&docsetVersion=14.3&locale=em and tried again, following the instructions. I set the "Decisions", and then connected it to the "Decision Tree", and on Train Properties, I defined Import Tree Model as Yes, runned the tree, and got an error

What I'm doing wrong?

How to use validate and test datasets manually in PROC GRADBOOST?

mh2t — Thu, 09 Jul 2020 15:43:20 GMT

I did split my dataset into 3 separate sas datasets train, validate and test.

I wanted to build a GBM model on the train set, check on the validate set and predict on the test set. How can I use the validate and test sets in my code explicitly for model checking and prediction?

proc gradboost data=mylib.train outmodel=mylib.savedModel seed=12345;
   input &myVars / level = nominal;
   target Y/ level = nominal;
   ods output FitStatistics=fitstats;
run;

Your help would be greatly appreciated!

Aircraft Turnaround Management Using Computer Vision

mentos05 — Mon, 06 Jul 2020 13:32:29 GMT

In this article I will demonstrate how Computer Vision can support the process of turnaround management in the aviation industry. Therefore I’ve developed a demo application and decided to share my thoughts I had during the development.

But first let’s clarify what turnaround management really is by learning from the experts:

Turnaround management, when applied to the aviation industry, refers to the physical process of preparing an aircraft for its next flight […]. The process of turnaround management is addressed when an aircraft is on the ground — from the time it arrives at a terminal gate, to the time it departs on its next scheduled flight.

-Collins Aerospace Airport Knowledge

In my simple words, turnaround management is everything that has to be done on the ground to prepare the aircraft for the next flight including boarding, baggage loading and fueling.

This process is crucial for every airline and should be optimized in every way possible to reduce downtimes, save costs and make customers satisfied with in time flights.

One of the main problems optimizing the process is the lack of actual knowledge about the current performance in the fleet. In this demo I show my way of using Computer Vision to extract this knowledge from camera streams at airports.

I am going to use different software products for different tasks:

SAS Viya to develop an object detection model
SAS Event Stream Processing to analyze the video in stream (also called SAS ESP)
Python interfaces from SAS (SAS SWAT, SAS DLPy, SAS ESPPy) to communicate with the SAS software
Open Source libraries like OpenCV, Numpy and matplotlib for analytical and visual tasks, e.g. polygon calculation and visualizing results

As you can see, this work is a combination of powerful SAS software and easy-to-use open source packages in Python. Why choose one if you can have both? 🙂

But now let’s get our hands dirty!

1. Define your business processes

Yes, everyone loves training computer vision models and visualize the results. But before starting your precious GPUs you have to decide what business processes you’re interest in and how they are defined.

I am by no means a turnaround expert so don’t expect too much from my definitions. You can define them as complex as you need them. For this demo I kept it simple, e.g. : If I have at least 2 people on the stairways I assume that the boarding has started.

Aircraft Turnaround Task Definitions

Defining these business processes will support you in the next two parts.

2. Define the objects you want to detect and train a model

Now, given the process definitions it will be easy to create a list of objects you want to detect. In my case I decided to train the model for the following objects: aircraft, person, baggage truck, ramp loader, bus, fuel truck, tank hose, ground power, stairway and rolling stairway

Yes these are more objects than I have in my process definitions but I thought I could maybe use some of them later. 🙂

If you’re unlucky like me and don’t have access to training data you can try to find videos on YouTube. For my example I used this video and labeled it using CVAT.

Finally, yes, now is the time for your GPU power!

For object detection I used a standard Tiny YOLO V2 model but other object detection models are possible. The training process is straight-forward but you can have a look at it here.

3. Define Areas of Interest

With your model being able to detect relevant objects you should now go back to your process definitions and extract the relevant areas for your business process. For example you don’t want to count the guys lifting your heavy baggage into the aircraft as passengers boarding the aircraft.

Looking at my definitions, I’ve decided to create the following five Areas of Interest: aircraft area, baggage area, stairway 1, stairway 2, fueling area

Areas of Interest and their corresponding Task Descriptions

For demo purposes I decided to manually define the Areas of Interest. However, you could of course define them dynamically given the objects you detect. This way the application would be more general, e.g. to fit other aircraft types or airports.

4. Create your Turnaround Management application

Now comes the fun part and we are putting all the pieces together to develop our Computer Vision aided Turnaround Management application.

As said in the beginning I’ve used SAS Event Stream Processing to create the image analysis process and afterwards connected this stream to SAS Viya to serve a simple SAS Visual Analytics dashboard showing a turnaround timeline.

From Streaming Data to Live Dashboarding

Turnaround Image Analysis in SAS Event Stream Processing

The SAS ESP process is very standard at the beginning. We are loading and deploying or computer vision model and provide it to the scoring window. Additionally we are receiving the camera image and resize it to fit the needs of the YOLO v2 model.

The YOLO v2 model is applied to the images and we receive the object coordinates. These object coordinates are then fed into a calculation window which will apply our process definitions and produce the visualizations.

Because I am a big fan of programming in Python, I decided to code the business rules in Python too. This is not a problem because the calculation windows in ESP allow us to run Python code via SAS Micro Analytics Services. There are even “PythonHelper Windows” in the SAS ESPPy interface to SAS ESP where you only have to provide the location of your code making the whole process very simple.

The calculation window produces one line of data per frame, including the scored image and several timestamp variables. I’ve used OpenCV to display the scored frames and also coded a small and ugly “dashboard”. Having this dashboard next to our image allows us to check whether all processes have been identified correctly.

Analyzing the video stream of a Turnaround - Visualized output of the SAS Event Stream Processing Pipeline

Turnaround Dashboard in SAS Visual Analytics

Even though looking at the results gives us already a feeling of being done, this is usually not the case. Applying computer vision models to your images is useless if you don’t make use of the information they extracted from images.

Therefore I decided to use the SAS Cloud Analytic Services Adapter to push the results into my SAS Viya environment. This allows me to use the data directly for further analysis, e.g. creating nice reportings or even do more sophisticated stuff like predictive modelling or forecasting.

All I had to do is to add a transpose window at the end of my SAS ESP process that creates the nice and small 3-column table that you can see in the diagram.

The rest is simple dashboarding as you would expect it from professional reportig and analysis software.

I am not a great report designer but you can have a look at my example report here:

Streaming the analyzed data to SAS Visual Analytics for Live Dashboarding

On the left side I used a so called “Data -Driven Content Object” that links to a simple websocket connector written in JavaScript to grab the scored frames from my ESP process. (Yes you can connect to ESP via a lot of different ways!) The good thing about a Data-Driven Content Object is that you can not only receive data but also send data to it directly from VA. This would enable us to interactively change the connection to a different ESP process that observes another aircraft.

The left side uses the transposed data coming from the ESP CAS adapter to create a simple timeline chart. The report refreshes its data every second to accurately show the current process status.

With the data being available in SAS Viya for all kinds of analysis, I’d like to finish this article and would love to hear about your thoughts.

Michael Gorkow | Data Scientist @ SAS Germany & CV-Enthusiast

LinkedIn | GitHub | Medium.com

Crime Investigation using Image Data in the times of Big Data

mentos05 — Mon, 06 Jul 2020 13:09:47 GMT

Social media platforms such as Facebook, WhatsApp and WeChat are wonderful ways to communicate with people all over the world.
Unfortunately these platforms are also used by criminals which is the reason why crime fighters want to investigate the contents on these platforms.

While these platforms have stored a lot of metadata which is more or less available (especially for government organizations), the most interesting content is of course the one that is created by the users. This content usually consists of unstructured data such as text, speech, videos and images. In this article I am focusing on the latter, images, but the approach itself can be used for all kinds of unstructured data.

Our task: Enabling efficient ways of investigating image data

Now, imagine being part of an investigation group of the police that is responsible to fight weapon crimes, right-wing or terroristic organizations.
You have scraped data from various channels, e.g. Twitter, Facebook, WeChat & Co. and now you’re sitting on a huge pile of images not knowing where to start looking.
Manually investigating these data sources is impossible, due to their size and variety.

Our goal is to provide an efficient way of filtering this data, allowing investigation officers to focus on the relevant images and their sources. This not only saves manual effort but also enables officials to better protect the law.

The idea: Image classification to create additional metadata

The idea is simple. We will train an image classification model that is able to classify our images into various categories. So to speak, we generate additional metadata for our images.

This classification is then combined with other data we already have such as source, date, location, etc.

Finally we want to create an intuitive dashboard that enables investigation officers to quickly identify potentially interesting content and skip through all the nice and funny images of cats. 🙂

As soon as we have the trained model available, the workflow is the following:

Metadata coming from new Images is directly fed into the analysis environment such as SAS Viya.
Images are scored using our computer vision model
Classifications and Metadata are combined

The combined data from step 3 is then fed into a visual report allowing crime investigators to perform in-depth analysis.

Getting our hands dirty: Training the image classification model

I am using the deep learning capabilities of SAS Viya to develop the image classification model. Of course you can develop the model using the SAS programming language but I am more comfortable using Python and Jupyter Notebooks. Luckily there are powerful Python-APIs that allow you to interact with your SAS environment. I am using the following APIs

SAS SWAT (communicate with SAS Viya)
SAS DLPy (special deep learning API for SAS)

In this Jupyter Notebook you can see how I use transfer-learning to train a ResNet-50 image classification model that afterwards can detect different weapons. The model was pretrained on the ImageNet dataset which contains images from 1000 classes and can be downloaded here.

There is also a second example where I apply the same process to images from hate symbols such as ISIS flags or swastikas.

The data used for training comes from two different sources:

Kaggle Weapon Dataset
Hate symbols that I scraped from Google Image search by hand

Here is an example how my training data looks like for the weapon detection:

The Jupyter Notebook comes with a lot of comments that explain in-depth what I am doing. Therefore I am not going into the details here to keep the article as short as possible.

After training the model, I evaluated its performance using a confusion matrix. As you can see the predictions are quite accurate with an accuracy of 99%. This seems a little bit too high for me and might be due to the training data being really nice to us. A lot of images have a nice white background which you cannot expect from real social media images. 🙂

Finished? No! A nice model is useless for our investigation officers.

A lot of articles about Computer Vision end when the model was trained and successfully scored some new data. However if you want your efforts to be useful you also have to think about how to put your models into the bigger picture. While SAS has advanced technology for this, e.g. the SAS Model Manager, I am not going into the details of production-ready applications here.

Instead, I’d like to demonstrate how a simple dashboard that uses our model’s predictions could look like. The dashboard was built with standard functionality of SAS Visual Analytics.

Example 1: Investigate Images of Weapons coming from Social Media

Example 2: Detect Hate Symbols on Images coming from Social Media

The only interesting part is the “Data-Driven-Content Object” that is responsible for displaying the selected images which you can see on the right side of the reports. This object dynamically receives the data selection from the other report elements. Behind the scenes it is using a very simple JavaScript code to retrieve and display the selected images.

As you can see in the videos our model acts as a supplier of additional metadata. It allows us to use the filter in the top-left corner to select the image categories we are interested in.

This use case does not only apply to crime investigation but is much more general.

Even though this article focused on identifying crime there are of course many other interesting use cases. Another idea could be that you want to identify your brand’s logo or products in social media posts. This way you could for example promote authentic user postings that highlight advantages of your products or you can try to give additional support if it is a customer complaining.

Do you have other use case ideas for this approach? I’d like to hear about them in the comments!

Michael Gorkow | Data Scientist @ SAS Germany & CV-Enthusiast

LinkedIn | GitHub | Medium.com

Tracking Social Distancing Using Computer Vision

mentos05 — Mon, 06 Jul 2020 13:11:02 GMT

Due to the recent pandemic, several governments have decided to implement restrictions for social distance. While most people follow these guidelines, there are still people who ignore them for various reasons.

The overall goal of this project is to identify objects (i.e., people) who are following social distancing guidelines and those who are not. This is accomplished in the following way:

Transform images and object coordinates into a two-dimensional map if a homography matrix was provided
Calculate real world distances between detected objects if a homography matrix was provided. Otherwise use distances from the image directly.
Detect crowds that exceed a specified parameter
Visualize the results on the camera image

Object Detection

The first step in the process is to detect people in images. A Tiny Yolo V2 model was used for this task, but any detection model would work. The decision to use Tiny Yolo V2 was mainly driven because of its ease to use in SAS.

Transforming the Image and Calculating Distance

The second step is to calculate the distances between all objects. While this may sound simple, it is more complicated if you think about it. Normally a camera does not provide a top view. Instead it is at some angle which leads to certain perspective. This perspective can be very important when you calculate real world distances in images. The following three pictures illustrate the transformation process.

The second image shows the results after transforming the image using a Homography Matrix. A rectangle was able to be formed, but the distances are incorrect due to the camera's perspective. The third image show the results after a transformation matrix was applied. A distance of 356 pixels on the image shrinks to 149 pixels while a distance of 62 pixels in the other direction are 83 pixels in reality.

Identifying Crowds

Now that we have detected people and transformed the distances between them we want to determine whether we have crowds in our image. In this demonstration, KD-Trees from Scipy was used to look up nearest neighbors given a detected person and a maximum radius. The following functions were used:

cKDTree to efficiently calculate nearest neighbors.
query_ball_tree to query the KDTree with a person and a given maxium radius.

Streaming Process

The streaming process uses SAS Event Stream Processing (ESP) and can connect to any video data source. ESP lets you define the process either graphically or programmatically via a Python Interface.

The following diagram illustrates the streaming process:

The top two boxes load the trained Tiny YOLO V2 model and provide the model to the scoring window. The scoring window receives images that are resized to appropriate dimensions (416x416 pixels in this case). The scoring window provides the detected persons and their corresponding x, y, width, and height values.

The last box utilizes the Python inside SAS Event Stream Processing to transform the coordinates given the homography matrix. Additionally, it uses Scipy to perform crowd detection.

Ressources & Results

For a more detailed look (incl. code) at Tracking Social Distancing Using Computer Vision, refer to this GitHub page.

And now let's view the results!

The left side shows the original camera image that was annotated by our streaming process. On the right you can see the top view produced by our homography matrix. Green colors show long distances, orange and red indicate small distances. Last but not least blue squares are drawn around crowds of three or more people.

Michael Gorkow | Data Scientist @ SAS Germany & CV-Enthusiast

LinkedIn | GitHub | Medium.com

VDMML sas code node not affecting output data

Ullsokk — Wed, 01 Jul 2020 08:40:17 GMT

I have a simple code snippet I want to run in the SAS code node in VDMML, that removes varaibles with given suffixes. Here is my code:

proc contents noprint data=&dm_data out = contents (keep=name) ;
run;


data contents;
	set contents ;
	if index(lowcase(name),'_dt') then drop=1 ;
	if index(lowcase(name),'_rk') then drop=1 ;
	if index(lowcase(name),'_dttm') then drop=1 ;
run ; 
	
proc sql;
	select name into: dropvars separated by ' ' from contents where drop = 1;
	quit;

data &dm_data;
	set &dm_data (drop=&dropvars);
run;

The proc contents and SQL seems to work, and I get no errors in the log. The log even implies that the final dataset has 1602 variables, down 7 from the original 1609, just as expected. But the column is still there in the results view. And in the next node, the column is still visible, and populated with data.

Am I using the wrong macro variable name for the data? Do I need to do a CAS action? or do I have to do some metadata update?

Ddos Attack

abhishekpandey — Fri, 26 Jun 2020 07:54:34 GMT

Which machine learning is best for detection of ddos (Distributed Denial of Service Attacks) attack ?

If there is options to select one of the following.

1). Performance evaluation of Botnet DDoS attack detection using machine learning.

2). Smart Detection: An Online Approach for DoS/DDoS Attack Detection Using Machine Learning

3). DDoS Attacks Detection Using Machine Learning Algorithms.

4). Machine Learning Based DDoS Attack Detection From Source Side in Cloud.

5). DoS Attacks Intrusion Detection Algorithm Based on Support Vector Machine.

Event-based sampling in Viya

ravnen55 — Wed, 24 Jun 2020 13:31:43 GMT

I have a bit of a weird problem:

I constructed a project in model studio in Viya, and used event-based sampling. The different models all had Roc/accuracy curves that looked something like this:

Great, that is what I expected.

Now I had some issues with modelmanager, and I had to reconstruct the project. Using the same dataset, the same models and again using event-based sampling, the curves now looked like this:

The probabilites were now around 95 % for an event=1, whick was in accordance with the distribution in the original dataset but NOT what i would expect after having done evet-based-sampling.

Obviously I must have done something different in the second project...But any ideas of WHAT box I might have ticked or what I might have otherwise done, that can cause this difference, would be much appreciated

thx

K-fold Cross-validation of Neural Networks

dcfroehlich_aol_com — Sat, 20 Jun 2020 05:56:42 GMT

My question concerns the use of K-fold cross validation for artificial neural networks (NN). Specifically, I want to know where the final NN model parameters come from? Were they obtained by a fit to the entire data set? Or, were they from a fit to one of the K-1 fold data sets used for training. The SAS documentation is not clear on this issue. Does anyone have an answer?

How do I subset a large dataset using a list of specific ID numbers?

sasgyro — Thu, 18 Jun 2020 21:08:39 GMT

I have a series of patients who have certain characteristics( variables)

I need to pick out the patients that meet the criteria by using their patient ID (300 unique identifiers) and separate them into their own dataset so they are alike only by that one characteristic.

i=I want it to be like this:

Data want;
SET have;
where patientnum= 3 6 5 4 8 7 9 0 ;
run;

but this doesn't work. and using OR and AND statements cancels the previous ID number

Text Mining

Ghabek — Thu, 18 Jun 2020 16:55:39 GMT

Hello All

Im using an old dataset from the text miner days in 9.4 to show subrogation using insurance adjustor notes to improve the model accuracy. There are some categorical variables but I cant find the mapping of the levels for the fields.....I guess I can make it up but would rather get the actual data definition if they actually existed!!

What should be the Optimum Number of Cluster

arpitsharma27 — Wed, 17 Jun 2020 17:26:16 GMT

Team,

I have this:

%macro clustering_method(method=);
proc cluster data=cars method=&method. ccc outtree=tree_&method. noprint;
where type="Sports";
by type;
	var horsepower mpg_highway weight wheelbase;
run;
proc sort data=tree_&method. out=&method.(keep= type _ncl_ _ccc_ );
by type _ncl_ _ccc_ ;
where not missing(_ccc_);
run;

%mend;
%clustering_method(method=Average);
%clustering_method(method=median);
%clustering_method(method=centroid);
%clustering_method(method=mcquitty);
%clustering_method(method=ward);

data Have;
set Average 
	Median 
	Centroid 
	McQuitty 
	Ward 
		indsname=source;
input_ds=scan(source,2,'.');;
run;

Referring to the Have dataset. What should be my optimum number of Clusters ? and Why?

Please advise.

Thanks

Gini performance metric for decision tree

Scott86 — Tue, 16 Jun 2020 23:52:38 GMT

Hi everyone,

I am building two classification decision tree in SAS enterprise miner. I'd like to compare the performance of the decision trees with a Gini metric. How do I get performance stats when using SAS enterprise miner? If I can't get Gini can I get ROC or AUC and then I can calculate Gini.

Thank you for any help

Miner interactive binning export

miner_girl — Fri, 12 Jun 2020 12:25:21 GMT

Dear SAS community,

I need help regarding export of interactive binning node, part of SAS EM.

I've split variables in groups and know I would like to observe their monotonic trend.

This view is very helpful and easy to explain to others. Now, I would like to export this view and then use it for presentation or for documentation. Do you have any idea how it can be managed?

I've tried reporter node, but it can draw only 25 variables and bucket's boundaries are below in table, therefore it makes it harder to interpret.

Thank you in advance for all suggestions.

Model comparison in SAS VS How does this work

PaalNavestad — Thu, 11 Jun 2020 13:56:40 GMT

Hi, I'm using the model comparison in SAS/VS. It does some strange things. I have an RF model with reported Validation ASE of 197,611. I have a Gradient Boosting (DSG) model with a reported Validation ASE of 199,686. After running this through model comparison I get a validation ASE of 196,927 for the RF model and 202,275 for the DSG model. These numbers vary with the number of models being in the Model Comparison.Random forest variable importanceGradientBoosting variable importanceModel comparison HeatmapModelcomparison table

Of a larger concern is that the Variable importance is completely changed. The variable that in the DSG model is the second most important has suddenly become the 7 most important after Model comparison. Also the variable importance HeatMap is inconsistent with the variable importance table.

SAS EM Leveraging - HP Text Miner + HP Forest

Ritwick — Mon, 08 Jun 2020 21:01:00 GMT

Hi Folks,

I am trying to build a churn model with the attached model flow. The flow worked well when using the test data. "HP Forest &SVD" node got selected in the Model Comparison. But when I am scored the selected model with the new dataset, the score node is only considering the HP Text Miner output. Let me know how I can resolve this issue.

Case when multiple condition in expression

David_Billa — Sun, 07 Jun 2020 12:50:00 GMT

I need help writing the case when with multiple conditions to use in 'expression' in DI Job. I'm getting an syntax error if I use below. How to keep adding the conditions in case when? Shouldn't we use 'else' to seperate the condition?

case when ENTITY_ID_SF in ('1000','1111')  and SEGMENT='NL' then (sum(GR_INPUT)/(sum(PREM_RES_GR)/3)) 
end
else
case when ENTITY_ID_SF in ('1000','1111') and SEGMENT='PL' then (sum(GR_INPUT)/(sum(PREM_RES_GR)/3)) 
end

Quick way to change the data type of registered table in SAS DI 4.9

David_Billa — Sat, 06 Jun 2020 20:30:33 GMT

I want to change the data type of one of the variable from character to numeric in one of the table in SAS DI 4.9. While I creating the new table, I had mistakenly created one variable with character data type instead of numeric.

So when I tried to change the datatype by going to respective table via inventory and changed the data type and did right click and choose 'Update Metadata' but still I see the datatype has not changed.

I ensured that I've closed the table before doing this operation. Am I doing something wrong here or is there any other way to change data type in a step or two?

ERROR: Local CASLIB quota exceeded.

klroesner — Fri, 05 Jun 2020 14:47:49 GMT

Hello World,

We are trying to run a random forest model in SAS Viya 3.4 VDMML and keep getting this error. Does anyone know what this means? And what is meant by 'quota'? We have not set any OS quota. And maybe one more question where the astore file _6WR5KOEIE2Y4J12PNBZEZAR3Y_AST is written to?

Thanks,

Klaus

NOTE: Writing HTML5 Body file: ods.htm
NOTE: Fileref _FR has been deassigned.
NOTE: The file _TFREF is:
      
      Dateiname=/opt/sas/viya/config/var/tmp/compsrv/default/3246bba6-2709-4bed-b816-b70ff22102a8/SAS_workDA4200000E54_rviap0v01.dst
      .tk-inline.net/74b72ccd-f517-4a62-a07e-53a6196492be/traincode.sas,
      Besitzername=p226526,Gruppenname=tk,
      Zugriffsberechtigung=-rw-r--r--,
      Zuletzt geändert=05. Juni 2020 15.13 Uhr,
      Dateigröße (Byte)=0
NOTE: 19 records were written to the file _TFREF.
      The minimum record length was 3.
      The maximum record length was 93.
NOTE:  Verwendet wurde: DATA statement - (Gesamtverarbeitungszeit):
      real time           0.00 seconds
      cpu time            0.01 seconds
      
NOTE: Using SEED=12345 for FOREST model building.
NOTE: Wrote 10139446832 bytes to the savestate file _6WR5KOEIE2Y4J12PNBZEZAR3Y_AST.
ERROR: Local CASLIB quota exceeded.
ERROR: The action stopped due to errors.

Using interactive grouping labels in custom SAS EM node

tom_evans_79 — Fri, 05 Jun 2020 04:13:59 GMT

I have an Enterprise Miner diagram which does the following: Inputs a dataset -> Partitions into training and validation -> Groups variables using Interactive Grouping -> Creates a credit Scorecard -> Scores up the dataset with the Score node -> Creates a custom report in a SAS Code node.

In the last node (SAS Code), I will calculate various statistics, but essentially I need to output a frequency table for grouped variables, using the group labels created by the Interactive Grouping node. The question is how I access those labels.

To illustrate, here's a very simple piece of code that I could use in the SAS Code node:

proc freq data=&EM_IMPORT_DATA;
	table GRP_age / norow nocol nopercent nocum missing;
run;

The variable 'GRP_age' is created by the Score node using the grouping definitions from Interactive Grouping, and is derived from the input variable 'age'.

This creates the following result:

GRP_age Frequency
--------------------
1 755
2 1190
3 1302
4 1580
5 1100
6 643

...but I would like to see this:

Group Name Frequency
----------------------------

age< 32, _MISSING_ 755
32<= age< 40 1190
40<= age< 47 1302
47<= age< 56 1580
56<= age< 65 1100
65<= age 643

Guessing I need to merge in a table from the IG node. Any ideas what this other table is called (in a way that will work if I connect my SAS Code node to a new Interactive Grouping node)?

Thanks!

Tom.

variable clustering dataset

Ghabek — Thu, 04 Jun 2020 18:19:03 GMT

Hello

I am still searching for a good dataset showcasing variable clustering in VDMML....maybe showing three or four clusters of variables to reduce multicollinearity on numeric variables.

any assistance would be appreciated!

variable clustering dataset

Ghabek — Tue, 02 Jun 2020 18:20:32 GMT

im looking for a good dataset to do variable clustering on.

WANTED - Your Insights

AnnaRhyne — Wed, 27 May 2020 13:19:44 GMT

We are regularly asked for stories about how our products have impacted our customers' business. Often they are looking to understand how our software contributes to making peoples lives better, whether that be for their customers or for you in being able to perform your job better, faster, and more efficiently.

Today, you can help provide insights for someone else's purchasing decision. Please take a moment to write a review on Gartner Peer Insights about your experience with SAS.

Reviews are anonymous and take around 10 minutes to complete. As thanks for your approved review, the first 50 reviewers will receive a $25 Visa gift card from Gartner.

Submit your review here.

We appreciate your feedback!

Free Webinar: How Do I Get Started with SAS® Visual Data Mining and Machine Learning?

MelodieRush — Tue, 26 May 2020 14:56:07 GMT

Hi Data Mining and Machine Learning Community,

I’m presenting a live “Ask the Expert” webinar on May 27, 11:00 – 12:00 p.m. ET How Do I Get Started with SAS^® Visual Data Mining and Machine Learning? Please join me.

About the webinar:

SAS Visual Data Mining and Machine Learning is designed for the data scientist, statistician and advanced business analyst. Whether you want to program or point and click, SAS Visual Data Mining and Machine Learning provides innovative algorithms and fast, in-memory processing. This session covers its capabilities and has an accompanying demonstration that provides a view into the components of SAS Visual Data Mining and Machine Learning

What you will learn

Interactive programming in a web-based development environment
Intelligent automation including Automatic Feature Engineering node for automatically cleansing, transforming, and selecting features for models.
Natural language generation
Embedded support for Python & R languages
Deep learning with Python (DLPy)
SAS® procedures (PROCs) & CAS actions
Highly scalable, distributed in-memory analytical processing

Are there any other specific questions you’d like covered? Let me know by responding to this thread.

Want more tips? Be sure to subscribe to the Ask the Expert board to receive follow up Q/A, slides and recordings from other SAS Ask the Expert webinars. To subscribe, select Subscribe from the Options drop down button above the articles.

Can't join the live event? You can view this and other Ask the Experts sessions on-demand here.

SAS enterprise miner decision tree splitting

Scott86 — Mon, 25 May 2020 03:16:37 GMT

Hi Everyone,

I am creating a decision tree and I want it to be only 3 nodes deep. I am controlling the splitting by using the leaf size under the Node area for Decision tree models, I am setting it to 200 (that is a minimum of 200 obs per leaf). The issue is the target I am splitting up which is binary (1 and 2) has a low volume. The target 2 only account for 0.44% of the observations. This is creating difficulty in the tree splitting. When I set the leaf size to 200 it goes to deep but when I increase it the tree does not split at all. I have tried changing the Significance level but it has no affect.

Nominal target Criterion is ProbChisq and significance level is 0.2.

Any help on how to tune the Hyper-Parameters to prune the model would be great.

Thanks

Scott

How to determine the one optimal decision threshold across multiple predictive models

JamieTee — Sun, 24 May 2020 03:10:13 GMT

I am currently wanting to construct predictive models to identify patients who may have a misdiagnosis of a certain disease. However, the issue is that I have multiple databases and I do not want to combine the data into one set, so I need to create predictive models for each database. On top of that, I wanted to use different model approaches and use the most accurate model.

I'm currently facing the problem of determining a way to identify the optimal decision threshold to categorize the results of the predictive model as having been misdiagnosed (p=1) or not having been misdiagnosed (p=0). I've read about how we can use Youden's Index to determine the optimal cutoff for a model, but since I'll have multiple models for each of the database, would it make sense to use the SAME cuttoff across all the models and databases or have one for each database (but keep it consistent for each model within the database)? I'm a bit lost on what the best approach is. I haven't been able to find papers that provide detail on how they determine optimal decision thresholds on multiple models at the same time

P.S. I've read about how you can use the cost of each result (i.e., TP, TN, FP, FN) to determine the threshold, but these values are unknown for my disease of interest.

Sorry about the long post, and thank you for your help in advance!

Your input is needed: Participate in the Rexer Analytics 2020 Data Science Survey

AnnaBrown — Thu, 21 May 2020 15:35:33 GMT

Hello community,

Rexer Analytics has been conducting the Data Science Survey since 2007. Each survey explores the analytic behaviors, views and preferences of data scientists and analytic professionals.

Make sure your perspective is included by taking the 2020 Survey! Your responses are completely confidential. A free report summarizing the findings will be available to everyone 4Q-2020.

Please share with your network of other data analysis professionals who may be interested.

Survey details:

— Link to Start the Survey (It takes about 20 minutes)

— Access Code: A2TF9

It is OK to share this Access Code with others, it can be used by multiple people.

— The Rexer Analytics main Data Science Survey page also has more survey information & FREE downloads of the 2007-2017 Survey Summary Reports

Error when running macro to generate TIME_DIM data

WorkingMan — Sun, 17 May 2020 10:06:45 GMT

Hi, in SAS Credit Scoring, there is TIME_DIM table under DIM library. From my understanding, there is a macro script that will load new month and year into this table when we execute. I want to load new years into this table and i managed to find the macro for this.

%bnkfdin;
%bankfdn_create_time_dim(yr_start=2004, yr_end=2016, mapto=);

When I run this in SAS Enterprise Guide, I get this error:

WARNING: Apparent symbolic reference FM_GRAIN not resolved.
ERROR: A character operand was not found in the %EVAL function or %IF condition where xxxxxxxxx
The condition was :
&FM_GRAIN ne 1 and &MapTo ne %sysfunc(dequote(&WEEK_START_FLG)) and &MapTo ne %sysfunc(dequote(&WEEK_END_FLG))

I have been trying to understand and browse through administrator guide for Credit Scoring, I still dont find any clue at all. In admin guide, I found this:

Set the correct value of the FM_GRAIN parameter in the
Custpara.Parameter table. Only numeric values are supported for this
parameter. Here are the values:
n 1 is Month (default value)
24 Chapter 2 / Installing SAS Credit Scoring
n 3 is Day

I have no idea what is custpara.parameter. Why is this SAS CS Admin Guide has such instruction that is so vague?

Is there anyone that can help me with this?

How to make a plot of residuals in survival analysis SAS EM

amsa1996 — Fri, 15 May 2020 09:59:37 GMT

Hi!

I have made a survival analysis in SAS EM using the Survival node (change time format). I want to detect anomalies/prediction errors in SAS EM from the survival analysis. Is it possible for me to make a plot of the residuals from the analysis in SAS EM? If yes, can someone help me with how i do this? If no, can someone help me with other options?

My target variable are categorical (0-1-2).

I know that i can use the export data ellipsis under "General" and then going on plot under "Actions". But then there are no predicted values such as "P_target" for my target variable that i can use.

Best regards,

Possible segment plot issue | RFM analysis | SAS Enterprise Miner

Vassilis — Wed, 13 May 2020 18:03:53 GMT

Hi all,

I am facing the following issue.. I am working on an RFM dataset previously extracted from SAS Enterprise Guide. Using the Cluster node, I keep getting this strange result on the Segment Plot where the Frequency variable shows a uniform distribution across clusters.

I exported the Frequency values for each segment and created respective number of histograms in Enterprise Guide, where it showed that there is not a uniform distribution in the values.

Is it then a bug from Enterprise Miner? Can I trust the analysis ignoring the Visualizer in the Segment Plot?

Thank you in advance for any guidance or information.

Enterprise Miner: What is "Selection Depth"?

PaigeMiller — Wed, 13 May 2020 14:05:55 GMT

As I am following this set of videos describing a modeling example in SAS Enterprise Miner, I see that the videos always use a "selection depth" of 5 to compare models. The documentation from SAS does not explain what "selection depth" is, it just says you can set it to 5 or 10 or some other number. Can someone explain what "selection depth" means (which, by the way, should be in the documentation somewhere) and why you might choose it to be the value of 5 rather than some other value?

SAS Enterprise Miner 15.1 and R 3.6.3

MariaD — Tue, 12 May 2020 14:30:08 GMT

Hi folks,

We just installed SAS EM 15.1 and R 3.6.3 on a new Linux server. Do you know which version of PMML and XML are compatible with SAS? I tested the integration with R through SAS EG and everything works fine.

I found information about PMML 1.4.2 and XML 3.98-1, but this versions are too old. The XML 3.98-1 require xml2-config that is not compatible with R 3.6.3. Anyone knows if the current version of PMML (2.3.1) and XML (3.99-0.3) is compatible with EM 15.1?

Regards,

EM 15.1 | R integration

MariaD — Tue, 05 May 2020 18:01:12 GMT

Hi folks,

We recently installed SAS EM 15.1 on our Linux Server. We already installed R 3.6.3 from Cran-project and we tested from SAS EG and it's working fine.

Which is the PMML version needed to integrate SAS EM with R?

Regards,

Re: Error When trying to create and run a model on a dataset in SAS Viya

TheAlchemist18 — Mon, 04 May 2020 16:04:58 GMT

Hi Madelyn_SAS, and thanks for your reply.
I did run similar table successfuly prior to this table and did not get any error massage. i.e. all tables that i run from now on give me an error. just got another error message when closing the previous one: "The advisor job for the project (ID <JOB NAME>) was not completed. The state of the job is "failed"."

ERROR: Connection failed. Server returned: Session reconnect failed: Could not find the specified

klroesner — Mon, 04 May 2020 12:43:50 GMT

Hello World,

We have a problem executing a CAS action: sequence.pathing . Does anyone know what the error message means? The program appears to be correct.

/*Dokumentation Action Set: Sequence Action Set */
/* https://documentation.sas.com/?docsetId=casanpg&docsetVersion=8.3&docsetTarget=n0gse7erokgd4un1gvqzehse1fkg.htm&locale=en*/


/* Libname */
cas seq;
libname mycas cas caslib=PowerUserDM;


/* What CAS sessions do I have running? */
cas _all_ list;


/* Zugriff auf die Daten über print */
proc print data=mycas.VP_SEQUENCE_DATA (obs=5);
run;


/*Reduktion der Daten für diesen Test */
data mycas.VP_SEQUENCE_DATA_tmp promote ; 
            set mycas.VP_SEQUENCE_DATA (obs = 10000); 
run;

/* Sequenzanalyse */
proc cas;                                                       
   sequence.pathing / table={caslib="PowerUserDM" name="VP_SEQUENCE_DATA_tmp"} 
                      item="EREIGNIS_ANONYM"
                      time="ZEIT"
                      tran="VS_VERS_NR_ANONYM"
                      casOutSequTranMap={caslib="casuser" name="casouttable", replace=true}
                      casout={caslib="casuser" name="versicherten_sequence", replace=true}
                      ;
   run;
quit;

cas seq terminate;

Log:

OPTIONS NONOTES NOSTIMER NOSOURCE NOSYNTAXCHECK;
75    
76    *----------------------------------------------------NICHT Lauffähiges Programm mit "proc cas pathing" auf artifiziellen
76  ! Daten----------------------------------------------------;
77    
78    
79    /*Dokumentation Action Set: Sequence Action Set */
80    /* https://documentation.sas.com/?docsetId=casanpg&docsetVersion=8.3&docsetTarget=n0gse7erokgd4un1gvqzehse1fkg.htm&locale=en*/
81    
82    
83    /* Libname */
84    cas seq;
NOTE: The session SEQ connected successfully to Cloud Analytic Services rviap0v02.dst.tk-inline.net using port 5570. The UUID is 
      544a6756-aa54-ad44-85c3-a8c0db933ac5. The user is P226526 and the active caslib is CASUSERHDFS(P226526).
NOTE: The SAS option SESSREF was updated with the value SEQ.
NOTE: The SAS macro _SESSREF_ was updated with the value SEQ.
NOTE: The session is using 4 workers.
85    libname mycas cas caslib=PowerUserDM;
NOTE: Libref MYCAS was successfully assigned as follows: 
      Engine:        CAS 
      Physical Name: 544a6756-aa54-ad44-85c3-a8c0db933ac5
86    
87    
88    /* What CAS sessions do I have running? */
89    cas _all_ list;
NOTE: Session SEQ is ACTIVE using port 5570 and host rviap0v02.dst.tk-inline.net for user P226526. The session UUID is 
      544a6756-aa54-ad44-85c3-a8c0db933ac5.
90    
91    
92    /* Zugriff auf die Daten über print */
93    proc print data=mycas.VP_SEQUENCE_DATA (obs=5);
94    run;
NOTE: The PROCEDURE PRINT printed page 2.
NOTE:  Verwendet wurde: PROZEDUR PRINT - (Gesamtverarbeitungszeit):
      real time           1.57 seconds
      cpu time            0.18 seconds
      
95    
96    
97    /*Reduktion der Daten für diesen Test */
98    data mycas.VP_SEQUENCE_DATA_tmp promote ;
99                set mycas.VP_SEQUENCE_DATA (obs = 10000);
100   run;
NOTE: There were 10000 observations read from the data set MYCAS.VP_SEQUENCE_DATA.
NOTE: The data set MYCAS.VP_SEQUENCE_DATA_TMP has 10000 observations and 3 variables.
NOTE: The data set WORK.PROMOTE has 10000 observations and 3 variables.
NOTE:  Verwendet wurde: DATA statement - (Gesamtverarbeitungszeit):
      real time           0.22 seconds
      cpu time            0.05 seconds
      
101   
102   /* Sequenzanalyse */
103   proc cas;
104      sequence.pathing / table={caslib="PowerUserDM" name="VP_SEQUENCE_DATA_tmp"}
105                         item="EREIGNIS_ANONYM"
106                         time="ZEIT"
107                         tran="VS_VERS_NR_ANONYM"
108                         casOutSequTranMap={caslib="casuser" name="casouttable", replace=true}
109                         casout={caslib="casuser" name="versicherten_sequence", replace=true}
110                         ;
111      run;
NOTE: Active Session now SEQ.
NOTE: Added action set 'sequence'.
ERROR: Connection failed. Server returned: Session reconnect failed: Could not find the specified session.
112   quit;
NOTE:  Verwendet wurde: PROZEDUR CAS - (Gesamtverarbeitungszeit):
      real time           46.16 seconds
      cpu time            1.89 seconds
      
113   
114   cas seq terminate;
NOTE: Libref MYCAS has been deassigned.
NOTE: Deletion of the session SEQ was successful.
NOTE: The default CAS session SEQ identified by SAS option SESSREF= was terminated. Use the OPTIONS statement to set the SESSREF= 
      option to an active session.
NOTE: Request to TERMINATE completed for session SEQ.
115   *----------------------------------------------------ENDE NICHT Lauffähiges Programm mit "proc cas pathing" auf artifiziellen
115 !  Daten----------------------------------------------------;
116   
117   OPTIONS NONOTES NOSTIMER NOSOURCE NOSYNTAXCHECK;
130

How to export a CNN model and use the output table with casl ?

rifcha — Sun, 03 May 2020 17:45:28 GMT

Hello,

i have build and trained a convolutional neutral betwork with CASL.

Then i have exported my model with the dlExportModel action set but i don't understant how to use the output table.

Can somone tell me how to use the output table in a CNN ?

Do i have to use the dlImportModelWeights action set ? if yes how to use it ?

The train model


proc cas;
	dlTrain / table={name='LargeImageDatashuffled', where='_PartInd_=1'} model='ConVNN' 
        modelWeights={name='ConVTrainedWeights_d', replace=1}
        bestweights={name='ConVbestweights', replace=1}
        inputs='_image_' 
        target='_label_' nominal={'_label_'}
        GPU=False
         ValidTable={name='LargeImageDatashuffled', where='_PartInd_=2'} 
        optimizer={minibatchsize=80, 
        			
        			algorithm={method='ADAM', lrpolicy='Step', gamma=0.6, stepsize=10,
       							beta1=0.9, beta2=0.999, learningrate=.01}
   			
        			maxepochs=1} 
        seed=12345
;

quit;

The export of my model




proc cas;
   dlExportModel /
      casout={name="ConVNN_save"}
      initWeights={name="ConVbestweights"}
      modelTable={name="ConVNN"};
run;


/*
Tables de sortie CAS
Bibliothèque CAS Nom
imagelib ConVNN_save
*/

SAS viya 3.5

Thanks for your help.

Prior Probabilities and Cumulative Lift

Shakir_Juolay — Thu, 30 Apr 2020 10:44:11 GMT

I would like to understand mathematically how the Cumulative Lift is adjusted for Prior Probabilities by SAS Enterprise Miner in the Regression Node for Logistic Regression.

I got the formula for adjusting posterior probabilities over here

https://documentation.sas.com/?docsetId=emxndg&docsetTarget=p1vqpbjwoo4bv7n1sw77e0z64xxs.htm&docsetVersion=15.1&locale=en

Test Data Performance

HASSRONA — Sat, 25 Apr 2020 18:37:01 GMT

In Model Comparison node, when test data results are better than training or validation results using logistic regression, what is this a sign of ?

SAS Enterprise Miner 15.1 very slow

MariaD — Tue, 21 Apr 2020 20:26:17 GMT

Hil Folks,

We just installed SAS 9.4 M6 and EM 15.1 on new Linux Server (Red Hat). Nobody is using the server yet. To test the SAS EM installation and execution, we created a simple EM project using Home Equity sample base. We added only the datasource node and a default tree node. The execution takes almost 2 minutes to run.

Any idea why is so slow?

Regards,

Importing a SAS EM HP Forest to RTDM

MaxLac — Fri, 24 Apr 2020 01:05:44 GMT

Hi everyone,

I've been struggling with this problem for quite a while now and I guess it's time to turn to the SAS Community.

Basically, I'm trying to use a Random Forest trained inside Entreprise Miner through RTDM. My team has tried different approaches (mainly the Two-Stage models as well as the GLM and the Gradient Boosting), but we're not getting the same level of performance with these models.

I can export the model Package and import it through Model Manager whithout any problem, but when it's time to use it inside RTDM, I get an error mentionning the fact that the model package does not have a generated DATA Step Score code.

The error makes sense since we see in the documentation that : "the SAS code that the HP Forest node generated during the run. HP Forest does not generate DATA Step score code. You must use PROC HP4SCORE to access HP Forest score code."

I understand these constraints and all, but has anyone tried/succeeded to use an HP Forest from SAS EM inside RTDM? If so, please, enlighten me. It might seem kinda strange, but it feels weird to me that we can't use EM-based model inside RTDM, I kinda felt like this was the whole point of our team using EM.

Thanks!

How to register a model using the macro %aa_model_register

Question — Wed, 22 Apr 2020 19:24:37 GMT

Hi,

I have a model built in SAS model Studio and would like to create a model package using %aa_macro_register. One of the parameter is scorecodefile (below). How should the path for . sas file look like?When I specify the path where my code is saved, it doesn't recognize the path.

I know how to register using the GUI but I would like to do it with the macro

Thank you

  %aa_model_register(
      modelname=&model_name1,
      modeldesc=%bquote(logselect_badloans model from CAS),
      data=mycas.&training_dataset,
      target=BAD,
      level=binary,
      scorecodefile=~/&model_name1..sas,
      scorecodeformat=DATASTEP,
      register=N,
      spk=Y,
      spkfolder=%str(~),
      miningfunction=classification,
      debug=N);

IMPORTANCE NVARS

segaqaci — Tue, 21 Apr 2020 03:05:30 GMT

Hello.

I have a gradient boosting tree where I want to get a list of all the variables used in the model. I currently have "IMPORTANCE NVARS=40",but is there a way to set this to ALL or MAX, so that the same code will work on any dataset? Or do I need to sum up the number of variables I have in the data set and put that into a macro variable?

Here is the code:

PROC TREEBOOST data=&abt._
CATEGORICALBINS=&CATEGORICALBINS.
INTERVALBINS=&INTERVALBINS.
EXHAUSTIVE=&EXHAUSTIVE.
INTERVALDECIMALS=&INTERVALDECIMALS.
LEAFSIZE=&LEAFSIZE.
MAXBRANCHES=&MAXBRANCHES.
ITERATIONS=&ITERATIONS.
MINCATSIZE=&MINCATSIZE.
MISSING=&MISSING.
LEAFFRACTION=&LEAFFRACTION. 
SEED=12345
SHRINKAGE=&SHRINKAGE.
SPLITSIZE=&SPLITSIZE.
TRAINPROPORTION=&TRAINPROPORTION.
;
INPUT &charvars. &flgvars. /level= nominal ; 
INPUT &numvars. /level= interval ; 
TARGET target_flg /level=binary ;
IMPORTANCE NVARS=40 OUTFIT=imp_VARS out = imp20 ;
SUBSERIES BEST ;
CODE FILE="&save_path.\&save_file." ;
SAVE MODEL=GBoost_Test  FIT=fit IMPORTANCE=imp  RULES=rules; 
RUN;

How do you deal with complex survey in Enterprise Miner

sammmy — Mon, 20 Apr 2020 12:28:26 GMT

I am working with a complex survey data, which has Strata, weight and PSU. How do I incorporate these structures in Enterprise Miner When building a machine learning model.

Percentage sum over training, validation and test datasets (EMiner)

CarlosP73 — Sat, 18 Apr 2020 21:48:33 GMT

Hi all,

I always wandered (but actually never checked until now) how EMiner (working with v15.1) deals with the percentages for tr, val and test ds, as you're allowed to enter numbers from 0 to 100 without any restriction at all.

I just tried today and entered 45%, 35% and 25% (adding up 105%) and surprisingly does not issue any error or warning

In order to finish my little test, I opened the number of records for each and below is what each ds actually got:

Tr(45%) -> 42.84%

Val(35) -> 33.32%

Test(25) -> 23.83%

Without any further calculation I just reckon it spreads the excess uniformly... but haven't found any documentation whatsoever...

Does anyone know what actually happens behind the scenes?

Thank you!

Carlos

[Anom] ifferences between positively skewed, reverse j-shape, negatively skewed and J-shape?

sareraes — Fri, 17 Apr 2020 17:02:49 GMT

I get a little bit confused when it comes to distinguishing the differences between them, For instance, J-shape and reverse j-shape just seem like a variant of negatively skewed and positively skewed.
What was a bit surprising to me was the fact that "number of major derogatory reports" and "the number of delinquent trade lines" plots are actually reverse j-shape distributions. Is this because they are extremely skewed on one end?

"years on current job" is positively skewed. But could you say this is reverse j-shaped also?

please see file attached

Filling In Values After Merging Two Datasets

Kemal_G — Fri, 17 Apr 2020 06:21:56 GMT

Dear All,

I am working on a large confidential healthcare data set. I am trying to make cost estimation based on it. In my dataset, there are cancer patients with survival info and their monthly cost. I will subtract patients' own pre-diagnosis monthly costs from post-diagnosis monthly cost to estimate cancer related cost I have postdiagnosis cost of patients until death (therefore number of months covered after diagnosis is different for each patients) but I have 12 month prediagnosis cost for each patients. I am planning to make subtraction based on the following logic:

Let us think about only one patient:

1. If this patient survives less than or equal to 12 months, I will subtract cost in each prediagnosis month from each postdiagnosis month:

cancer cost(i)=postdiagnosiscost(i)-prediagnosis cost(i), i=1,...,S S=number of months survived.

2. If patient survives more than 12 months, I will subtract cost in each prediagnosis month from each postdiagnosis month according to mod 12:

cancer cost(i)=postdiagnosis cost(i)-prediagnosis cost(j), i=1,..,S j=mod(i), S=number of months survived.

Let the patient live 26 months:

For cancer cost at 15th month I will have:

cancercost(15)=postdiagnosis cost(15)-prediagnosis cost(3),

For cancer cost at 25th month I will have:

cancercost(25)=postdiagnosis cost(15)-prediagnosis cost(1)

I merged these two datasets (prediagnosis and postdiagnosis cost) but I have missing values because I have only 12 months of prediagnosis cost and postdiagnosis cost for more than 12 months. I want to fill in missing values of prediagnosis cost by replicating them according to mod of survival of patients.

Can you help me in writing macro or code of this problem?

Thank you.

Enterprise Miner Interactive Decision Tree

Matim — Thu, 16 Apr 2020 21:39:06 GMT

Hello,

This is my first time accessing SAS communities and I have a question about a specific section of the software I'm using called Enterprise Miner. In the predictive analysis workspace, I've created three nodes and am working on the fourth node called "Decision Tree". When I create the decision tree and press the "..." button next to Interactive to open the "Interactive Decision Tree Application", I get an error message saying "Run Time error was encountered. Please see the log in the node results window for more details". What's causing this error to occur?

Thanks

How to import SAS Base models into Model Manager?

Question — Tue, 14 Apr 2020 12:38:19 GMT

Hi All,

Hope you are well.

I would like to know how you import SAS Base models into Model Manager?

I have imported models from Enterprise Miner in the past but was wondering if it's possible to import models done in Base SAS?

Thank you so much

SASMiner - Cannot connect to "SASMiner - Logical Workspace Server"

Jens89 — Tue, 14 Apr 2020 08:40:57 GMT

Hi,

When opening any of my projects in Miner, I keep on getting the following error:

SASMiner - Cannot connect to "SASMiner - Logical Workspace Server" Make sure the workspace server is running and accessible.

I have added a screenshot.

Why would SAS not Transform a variable?

BearerofSAS — Sat, 11 Apr 2020 17:33:30 GMT

As far as I understand, we transform variables to make them more symmetrical/linear relationship with the target variable and this can only be on interval data. But why might SAS choose not to transform a variable as opposed to transforming other explanatory variables (bearing in mind if they are all interval level data types)? Could it be because they might be already approximately symmetrical or the variable potentially causing overfitting? Hence they aren't transformed?

Cheers

SAS Model Studio - Cluster node

prajnat — Wed, 08 Apr 2020 14:32:36 GMT

Hi Team,

Following is the sample data with just 6 columns and 96 rows -

Emp_ID actually needs to be rejected and the variable Q1 - Q5 are inputs

Steps I have tried -

1. To perform any step in the pipelines the model studio is prompting me to define a target variable, in real picture I don't have a target variable. But I tried to create a dummy target variable with no values in it and this threw an error.

2. Just to see the performance of the cluster node, I provided Emp_ID itself as the target and ran the pipeline

3. My output had all the items I was looking for but I am confused as to why the segment plot was performed only on 57 variable while expected segment plot on all 96 rows. Is this due to data partition settings ? if yes, please let me know to edit the window and also confirm if this is a global setting change or the changes I make will be only to specific project

I need to see the segment plot on all the 96 rows.

Please revert

Thank you

Prajna

END + value in SAS CASE

duttarakhi89 — Tue, 07 Apr 2020 20:58:21 GMT

I need some urgent help in sas . stuck with SAS CASE with this structure .. CASE WHEN variable<= case when condition then when condition then ......... when condition then else end + (numeric value) then .. again starting from outer WHEN CASE variable= case when condition then when condition then ......... when condition then else end + (numeric value) then .. its a big nested kind of ..

what does it mean by end + 3 / end + 6 after inner case ending ... can anyone please help to explain the concept behind end + value

What’s the differences between positively skewed, reverse j-shape, negatively skewed and J-shape?

BearerofSAS — Sat, 04 Apr 2020 01:47:42 GMT

I get a little bit confused when it comes to distinguishing the differences between them, For instance, J-shape and reverse j-shape just seem like a variant of negatively skewed and positively skewed.

What was a bit surprising to me was the fact that "number of major derogatory reports" and "the number of delinquent trade lines" plots are actually reverse j-shape distributions. Is this because they are extremely skewed on one end?

"years on current job" is positively skewed. But could you say this is reverse j-shaped also?

please see file attached

Cheers

Results of the decision tree does not show a clear split between churn and non churn. why?

Ann_90 — Fri, 03 Apr 2020 03:23:23 GMT

Details about the data source: credit card churn data

It has only 5% churn data. Because of this rare event, tried changing the decision weights at the data source.

Here is a screenshot of the tree:

I have taken care of the quasi-separation/separation issues by excluding them from the metadata node which is then connected to the decision tree node.

1. However, the results does not show a proper split between churn and non-churn. How to overcome this problem? Kindly help. Thanks

2. Also, wanted to check if choose to change the decision weights at the data source, should the assessment measure in the decision tree properties(panel) only be 'DECISION'? or can I use other assessment measures? What does the LIFT measure mean? I noticed that using the lift measure helped to improve the distribution of the churn and non-churn in my best nodes. But am not sure what the LIFT measure means.

3. Under the split search option for decision trees, should 'use decision' be set to Yes since decision weights were selected at the data source?

Kindly help with the above doubts! really appreciate it!

Configuration venv Python for SASModelStudio

Mena11 — Tue, 31 Mar 2020 13:11:08 GMT

Hi everyone,

I open the following track to ask for clarifications on the configuration of the node "Open source code" in SASModelStudio in Viya 3.5
In particular, I was wondering if it was possible to set the use of a specific virtualenv (in Python) based on the type of user.

At the beginning of the user session it would be necessary to invoke the command "source bin/activate" of a certain venv.

[user1@machine /home/user1/venv]# source bin/activate
(venv) [user1@machine /home/user1/venv]#

If it wasn't possible to discriminate by user/group, could it be possible to set a default venv in some configuration file?

Thanks

Davide

Difference between variable importance and feature engineering in model studio

prajnat — Tue, 31 Mar 2020 06:42:03 GMT

Hi Team,

I was interested in getting mutual information for variables in a churn model. I see that under feature engineering we have PCA,SVD, auto encoder etc.. I would like to know what could be the best information criteria to use instead of mutual information

Analysis by Deciles in Model Studio

Pmyosh — Tue, 31 Mar 2020 02:56:28 GMT

Hi All,

I am very new to Model Studio, is there any way I can see the results by deciles?

Previously, when models are built using the manual way in SAS EG, data was split to deciles whereby the highest decile has the highest propensity.

This is very useful when doing the tracking after the model deployment.

Or is there any way to do such trackings using the Model Studio? Thanks a lot!

Can I combine predictive modelling with time series?

Adeline — Mon, 30 Mar 2020 22:10:46 GMT

I need to build up a model to predict sales using several variabls (including info about date, promo, competition, etc.). How can I combine the pattern in date (time series) and other input variables in SAS miner?

Another question is about partitioning the data into training and validation. How can I partition the data while preserving the patterns of time series? I've seen some articles on SAS Miner website. Some articles say I can do that only by using SAS code (which is a little difficult for me to understant). Others say I can do that with the partition node's user-defined sampling. But I do not see "user-defined" but just "cluster" sampling. Are they the same?

Hope those experts could help me out of this. Thanks!

SAS Model Studio - Confusion matrix

prajnat — Mon, 30 Mar 2020 03:54:47 GMT

Hi Team,

I am new to sas model studio and was looking for confusion matrix to find the TP FP FN FP and the associated results.

I remember it is very simple in sas eminer and the results were more customized. I am unable to find the confusion matrix on model studio.

Please let me know if anyone has worked on the similar situation before or have a solution for the same.

Thank you,

Prajna

Vertical Line for recommended cluster size in CCC plot in Enterprise Miner

SASEnthusiast1 — Sat, 28 Mar 2020 09:48:49 GMT

Hello,

I am new to using SAS Enterprise Miner Client 13.2. I would like to know to get thin blue vertical line which SAS displays in CCC plots to appear. This would be the optimal, recommended cluster size from CCC plots in Enterprise Miner.

When I go to View > Summary Statistics > CCC plot, I was able to make a successful CCC plot but SAS just does not recommend any optimal cluster size. In essence, the vertical line depicting the optimal cluster size solution does NOT appear.

Please see output CCC plot attached. I know there is a big jump around the sample size of 32 in the plot but I want SAS to automatically recommend a cluster size.

Any help would be greatly appreciated. Many thanks in advance!

ERROR: File WORK.CARDATATRAIN.DATA does not exist

azmol — Fri, 27 Mar 2020 14:47:27 GMT

I'm running some code in SAS Enterprise Miner 15.1 from a SAS blog:

proc sgscatter data=CARDATA_TRAIN;
  label make='Make' weight='Weight';
  label horse_power='Horse Power' mpg='mpg';
  matrix make weight horse_power  mpg;
  run;

I'm getting the error

ERROR: File WORK.CARDATATRAIN.DATA does not exist.

The error is correct as the SAS table is called CARDATA_TRAIN, but SAS has prefixed the table name with "WORK" and suffixed with "DATA". Which is why I'm getting the error!

Why is SAS doing this? How do I over come this? Any advice would be appreciated.

SAS Enterprise Miner 14.1 Metadata Node Question

aami — Fri, 27 Mar 2020 04:35:54 GMT

In the properties node of the metadata node I can set the Advanced Advisor option to Yes. But I cannot access the Advanced Advisor Options dialog box from the metadata node. Is it not possible to do so? Is the Options dilog box accessible only from the data source node?

Thanks

Run Error Run Error "IO.OPEN" when trying to run StatExplore after creating target variable

PhilMStudy2020 — Wed, 18 Mar 2020 21:10:13 GMT

When I try to run the StatExplore node with a target variable designated in file import (a binary character target variable), I receive a "IO.Open" error code.

I do not receive this code at all when I do not use a target variable. All of the data is qualitative data

I did see another post on this from 2015 but there wasn't a solution.

How can we do multiple target/dependent regression model in SAS Studio or SAS Viya

Thimmy — Thu, 19 Mar 2020 11:57:20 GMT

I am trying to build regression model in SAS Viya VDMML through its two components which is SAS Studio and Build Models Section. The model which I would like to build has more than 1 target/dependent variable. I am facing a problem in both SAS Studio and Build Models Section, it is not allowing me to take more than 1 dependent/target variable. I have two questions:-

1. Can we build a model with more than 1 target/dependent variable in SAS Viya VDMML?Yes/No

2. If we can build the model, what are the ways to activate the feature? Is there any options which I need to enable it?

Assessing my Scorecard in SAS Miner

KevinL — Wed, 11 Mar 2020 16:08:45 GMT

Hi all,

I've recently started learning how to develop scorecards using Miner and have much to learn. For my most recent project, I've created a scorecard where the target is whether or not a news website publishes a story based on a limited number of variables.

In the Interactive Grouping Node, my variables have information values between 0.7 and 0.9. My Scorecard gives the below fit statistics and charts. From what I can see, the Misclassification Rate, KS, Gini and Area under the ROC look good. However, looking at the KS chart, I'm not sure what to make of it as I'm used to the curve resembling a normal distribution. I'm struggling to interpret this and more importantly to determine the cutoff score I need to use. Finally, I'm not sure if what the Classification Chart shows is normal as it appears the have a high rate of false positives (hope I'm interpreting this correctly).

If someone can assist I'd be most appreciative!

Evaluation problem using Start group, End Group nodes - Cross Validation

baabed — Tue, 10 Mar 2020 17:08:54 GMT

Hello everyone,

I'm running the below flow in SAS Enterprise Miner Version 15.1 with the purpose of running a semi-automatic hyper-parameter tuning using cross-validation and comparing the CV error of each parameters combination to select the best model within the logistic regression model for example.

I encounter two problems:

First: running each Model Import seperately runs smoothly as seen through the green ticks.

Running however the model comparison leads to the issue of the first model import flow not running as it stops at the stepwise node.

Second: The results of End Group do not include the appropiate score ranking measures like cumulative lift or gain etc but instead the predicted mean. Does anyone know why this is the case?

Any help is appreciated!

Thank you.

EEG brain data survival mining

pink_poodle — Tue, 10 Mar 2020 16:18:10 GMT

I have EEG time data from two groups of people - survivors and deaths. I would like to make a rule for what differentiates the signals in these two groups. My tools are SAS Enterprise Guide and SAS Enterprise Miner. I thought to use the following approach:

1. Separate time series into two groups in Miner using various available approaches

2. Score how well those groups correspond to actual groups of survivors and deaths

The first point seems like a fishing expedition. Can anyone with more experience suggest a tactics?

Can the second point be done in Miner? How?

All suggestions are welcome. Thank you very much for your help.

Accessing previous Diagram on Enterprise Miner

Kosi — Thu, 05 Mar 2020 13:53:44 GMT

I created a project a couple of days ago, but now I can not view the diagram. There is a pop up that says that the diagram is locked and that i need to close the diagram or delete the lock file. It will not allow me to close the diagram, but how do you delete the file lock?

ERROR:Table is too large to process. System error returned 3000. (Start Group, End Group,Cross Val)

baabed — Thu, 05 Mar 2020 14:12:43 GMT

Hi everyone,

I'm trying to run random forest models & cross validating on samples from the training set in SAS Enterprise Miner 15.1 using the "Start Groups" and "End Groups" nodes.

Every time I run the flow I receive an error that says "Run time error was encountered."

When checking the log the first error I find is "ERROR: The requested table is too large to process." and after that step "NOTE: The SAS System stopped processing this step because of errors." Does anyone have an idea how to resolve this issue?

Would appreciate any help!

Thank you in advance.

*------------------------------------------------------------*
Date:                March 05, 2020
Time:                14:30:30
Site:                70212334
Platform:            X64_SR12R2
Maintenance Release: 9.04.01M6P110718
EM Version:          15.1
* 
*------------------------------------------------------------*
* Training Log
Date:                March 05, 2020
Time:                14:30:27
*------------------------------------------------------------*
15231  proc freq data=EMWS4.Grp_VariableSet noprint;
15232  table ROLE*LEVEL/out=WORK.GrpMETA;
15233  run;
 
NOTE: There were 15 observations read from the data set EMWS4.GRP_VARIABLESET.
NOTE: The data set WORK.GRPMETA has 5 observations and 4 variables.
NOTE: PROCEDURE FREQ used (Total process time):
      real time           0.03 seconds
      cpu time            0.03 seconds
 
 
15234  proc print data=WORK.GrpMETA label noobs;
15235  var ROLE LEVEL COUNT;
15236  label ROLE = "%sysfunc(sasmsg(sashelp.dmine, meta_role_vlabel, NOQUOTE))" LEVEL = "%sysfunc(sasmsg(sashelp.dmine, meta_level_vlabel, NOQUOTE))" COUNT = "%sysfunc(sasmsg(sashelp.dmine, rpt_count_vlabel, NOQUOTE))";
15237  title9 ' ';
15238  title10 "%sysfunc(sasmsg(sashelp.dmine, rpt_varSummary_title  , NOQUOTE))";
15239  run;
 
NOTE: There were 5 observations read from the data set WORK.GRPMETA.
NOTE: The PROCEDURE PRINT printed page 1.
NOTE: PROCEDURE PRINT used (Total process time):
      real time           0.01 seconds
      cpu time            0.01 seconds
 
 
15240  title10;
 
15241  %let EMEXCEPTIONSTRING=;
PERFORMANCE  DETAILS
15581  *------------------------------------------------------------*;
15582  * Grp: Generation of macros and macro variables;
15583  * To see the code generated, set the EM_DEBUG macro variable to SOURCE or _ALL_;
15584  *------------------------------------------------------------*;
 
15585  %let EMEXCEPTIONSTRING=;
15586  *------------------------------------------------------------*;
15587  * TRAIN: Grp;
15588  *------------------------------------------------------------*;
15589  %let EM_ACTION = TRAIN;
15590  %let syscc = 0;
15591  %macro main;
15592
15593     filename temp catalog 'sashelp.emutil.gp_macros.source';
15594     %include temp;
15595     filename temp;
15596
15597     %SetProperties;
15598
15599     %if %upcase(&EM_ACTION) = CREATE %then %do;
15600
15601         filename temp catalog 'sashelp.emutil.gp_create.source';
15602         %include temp;
15603         filename temp;
15604
15605         %create;
15606     %end;
15607     %else
15608     %if %upcase(&EM_ACTION) = TRAIN %then %do;
15609
15610         filename temp catalog 'sashelp.emutil.gp_train.source';
15611         %include temp;
15612         filename temp;
15613
15614         %train;
15615     %end;
15616     %else
15617     %if %upcase(&EM_ACTION) = SCORE %then %do;
15618
15619         filename temp catalog 'sashelp.emutil.gp_score.source';
15620         %include temp;
15621         filename temp;
15622
15623         %score;
15624     %end;
15625     %else
15626     %if %upcase(&EM_ACTION) = REPORT %then %do;
15627
15628         filename temp catalog 'sashelp.emutil.gp_report.source';
15629         %include temp;
15630         filename temp;
15631
15632         %report;
15633
15634     %end;
15635     %else
15636     %if %upcase(&EM_ACTION) = POSTLOOP %then %do;
15637
15638         filename temp catalog 'sashelp.emutil.gp_postloop.source';
15639         %include temp;
15640         filename temp;
15641
15642         %postloop;
15643
15644     %end;
15645
15646     %doendm:
15647  %mend main;
15648
15649  %main;
NOTE: %INCLUDE (level 1) file TEMP is file SASHELP.EMUTIL.GP_MACROS.SOURCE.
15650 +%macro SetProperties;
15651 +   %em_checkmacro(name=EM_PROPERTY_MODE,            global=Y, value=STRATIFY);
15652 +   %em_checkmacro(name=EM_PROPERTY_TARGETGROUP,     global=Y, value=Y);
15653 +   %em_checkmacro(name=EM_PROPERTY_INDEXCOUNT,      global=Y, value=10);
15654 +   %em_checkmacro(name=EM_PROPERTY_MINIMUMOBS,      global=Y, value=10);
15655 +   %em_checkmacro(name=EM_PROPERTY_SIZETYPE,        global=Y, value=PERCENT);
15656 +   %em_checkmacro(name=EM_PROPERTY_SIZEOBS,         global=Y, value=.);
15657 +   %em_checkmacro(name=EM_PROPERTY_SIZEPERCENT,     global=Y, value=10);
15658 +%mend SetProperties;
15659 +
NOTE: %INCLUDE (level 1) ending.
NOTE: Fileref TEMP has been deassigned.
NOTE: %INCLUDE (level 1) file TEMP is file SASHELP.EMUTIL.GP_TRAIN.SOURCE.
15660 +%macro scoreValidTest(data=, out=);
15662 +   %if ("&data" ne "") and (%sysfunc(exist(&data, VIEW)) or %sysfunc(exist(&data))) %then %do;
15663 +       data &out / view = &out;
15664 +          retain _RESAMP_ 1;
15665 +          set &data;
15666 +       run;
15667 +   %end;
15669 +%mend scoreValidTest;
15671 +%macro getVariables;
15672 +   %if "&emloopnumber" = "1" %then %do;
15673 +       data _null_;
15674 +          retain numgoup 0;
15675 +          set &em_data_variableset end=eof;
15676 +          %if &em_num_target = 1 %then %do;
15677 +              %if &EM_PROPERTY_MODE = BOOSTING %then %do;
15678 +                  where (role = 'TARGET' and level ^= 'INTERVAL' and grouprole in('BOOST', 'DEFAULT'));
15679 +                  if role eq 'TARGET' then call symput('emboostvar', strip(NAME));
15680 +              %end;
15681 +              %else
15682 +              %if &EM_PROPERTY_MODE = BAGGING %then %do;
15683 +                  where (role = 'TARGET' and grouprole in('BAG', 'DEFAULT'));
15684 +                  if role eq 'TARGET' then call symput('embagvar', strip(NAME));
15685 +              %end;
15686 +          %end;
15687 +          %else %do;
15688 +              where (grouprole= 'GROUP' or (role = 'TARGET' and grouprole in('BAG', 'BOOST')) or (role = 'SEGMENT' and grouprole = 'DEFAULT'));
15689 +              if grouprole eq 'BAG' then call symput('embagvar', strip(NAME));
15690 +              else
15691 +              if grouprole eq 'BOOST' then call symput('emboostvar', strip(NAME));
15692 +          %end;
15693 +          if grouprole = 'GROUP' or (role = 'SEGMENT' and grouprole = 'DEFAULT') then do;
15694 +             call symput('emgroupvar', strip(symget('emgroupvar'))!!' '!!strip(NAME));
15695 +             numgroup + 1;
15696 +          end;
15697 +          if eof then call symput('emnumgroupvar', strip(put(numgroup, best.)));
15698 +       run;
15699 +    %end;
15700 +    %else %do;
15701 +          %if "&EM_PROPERTY_MODE" = "STRATIFY" or "&EM_PROPERTY_MODE" = "CROSSVALIDATION"  %then %do;
15702 +            data _null_;
15703 +               retain numgoup 0;
15704 +               set &em_data_variableset end=eof;
15705 +               where (grouprole= 'GROUP' or (role = 'SEGMENT' and grouprole = 'DEFAULT'));
15706 +               call symput('emgroupvar', strip(symget('emgroupvar'))!!' '!!strip(NAME));
15707 +               numgroup + 1;
15708 +               if eof then call symput('emnumgroupvar', strip(put(numgroup, best.)));
15709 +            run;
15710 +          %end;
15711 +            %else
15712 +        %if &EM_PROPERTY_MODE = BOOSTING %then %do;
15713 +            %em_getName(key=BOOSTINFO,   type=DATA);
15714 +            %if %sysfunc(exist(&em_user_boostinfo)) %then %do;
15715 +                data _null_;
15716 +                   set &em_user_boostinfo;
15717 +                   call symput('emboostvar', strip(_TARGET_));
15718 +                run;
15719 +            %end;
15720 +        %end;
15721 +        %else
15722 +            %if &EM_PROPERTY_MODE = BAGGING %then %do;
15723 +                %em_getName(key=BAGINFO,   type=DATA);
15724 +                %if %sysfunc(exist(&em_user_baginfo)) %then %do;
15725 +                    data _null_;
15726 +                       set &em_user_baginfo;
15727 +                      call symput('embagvar', strip(_TARGET_));
15728 +                    run;
15729 +                %end;
15730 +            %end;
15731 +    %end;
15732 +%mend getVariables;
15734 +%macro gp_bag(targetname=);
15735 +   %if %index(&EM_DEBUG, _ALL_) %then %do;
15736 +        %put LOOP MODE:   &emloopmode ;
15737 +        %put LOOP NUMBER: &emloopnumber;
15738 +        %put BAGGING TARGET: &targetname;
15739 +   %end;
15741 +   filename temp catalog 'sashelp.emutil.gp_boostmacros.source';
15742 +   %include temp;
15743 +   filename temp;
15745 +   %em_getName(key=BAGINFO,    type=DATA);
15746 +   %em_getName(key=EMLOOPINFO, type=DATA);
15748 +   %if &emloopnumber = 1 %then %do;
15749 +       filename gpDelta "&EM_FILE_CDELTA_TRAIN";
15750 +       data _null_;
15751 +          file gpdelta;
15752 +          put "if ROLE = 'TARGET' and NAME ne '&targetName' then delete;";
15753 +          put "else if upcase(NAME) eq '_RESAMP_' then do;";
15754 +          put "     LABEL='Bootstrap Frequency'; LEVEL='INTERVAL'; ROLE='FREQ';";
15755 +          put "end;";
15756 +       run;
15757 +       filename gpdelta;
15759 +       %if %sysfunc(exist(&em_user_baginfo)) %then %do;
15760 +           proc datasets lib=&em_lib nolist;
15761 +              delete &em_nodeid._baginfo;
15762 +           run;
15763 +       %end;
15765 +       %if "%em_freq" ne "" %then %do;
15766 +           proc means data=&em_import_data sum;
15767 +              var %em_freq;
15768 +              output out = _tempds sum=sumfreq;
15769 +           run;
15770 +           quit;
15771 +           data _null_;
15772 +              set _tempDs;
15773 +              call symput('_sumfreq', put(sumfreq, best12.));
15774 +           run;
15775 +       %end;
15776 +       %else %do;
15777 +           proc sql noprint;
15778 +              select count(*) into :_sumfreq
15779 +              from &em_import_data;
15780 +           quit;
15781 +       %end;
15783 +       %if "&em_property_SizeType" eq "PERCENT" %then %do;
15784 +           data _null_;
15785 +              call symput('_size', put(round(&_sumfreq*&em_property_sizepercent/100), best12.));
15786 +           run;
15787 +       %end;
15788 +       %else
15789 +           %let _size = &em_property_sizeobs;
15791 +       data &em_user_baginfo;
15792 +          length _LOOP_ 8 _TARGET_ $32 _SUMFREQ_ 8 _SIZE_ 8;
15793 +          _LOOP_     =&EM_PROPERTY_INDEXCOUNT;
15794 +          _TARGET_   ="&targetname";
15795 +          _SUMFREQ_  =&_sumfreq;
15796 +           _SIZE_    = &_size;
15797 +          output;
15798 +       run;
15799 +       data &em_user_emloopinfo;
15800 +          length MODE $8;
15801 +          label MODE     = "%sysfunc(sasmsg(sashelp.dmine, rpt_mode_vlabel,       NOQUOTE))"
15802 +                _LOOP_   = "%sysfunc(sasmsg(sashelp.dmine, rpt_groupIndex_vlabel, NOQUOTE))"
15803 +               _TARGET_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_target_vlabel,     NOQUOTE))";
15804 +           set &em_user_baginfo(keep=_LOOP_ _TARGET_);
15805 +           MODE='Bagging';
15806 +       run;
15808 +       data &em_data_eminfo;
15809 +          length TARGET KEY $32  DATA $43;
15810 +          TARGET=''; KEY='BAGINFO'; DATA="&em_nodeid"; output;
15811 +       run;
15812 +   %end;
15813 +   %else %do;
15814 +      data _null_;
15815 +         set &em_user_baginfo;
15816 +         call symput('_sumfreq', put(_sumfreq_, best12.));
15817 +         call symput('_size', put(_size_, best12.));
15818 +      run;
15819 +   %end;
15821 +   %let bagsumFreq  = &_sumFreq;
15822 +   %let bagSize     = &_Size;
15824 +   %embag( _tra=&em_export_train, _train=&em_import_data, _seed=&em_property_randomseed, _freq=%em_freq,
15825 +           _sumfreq=&bagsumfreq, _size=&bagsize, _loop=&emloopnumber);
15827 +   %scoreValidTest(data=&em_import_validate, out=em_export_validate);
15828 +   %scoreValidTest(data=&em_import_test,     out=em_export_test);
15830 +   %doendbg:
15832 +%mend gp_bag;
15835 +%macro gp_boost(targetname=);
15836 +   %if %index(&EM_DEBUG, _ALL_) %then %do;
15837 +       %put LOOP MODE:   &emloopmode ;
15838 +       %put LOOP NUMBER: &emloopnumber;
15839 +        %put BAGGING TARGET: &targetname;
15840 +   %end;
15842 +   filename temp catalog 'sashelp.emutil.gp_boostmacros.source';
15843 +   %include temp;
15844 +   filename temp;
15846 +   %em_getname(key=EMLOOPINFO, type=DATA);
15847 +   %em_getname(key=BOOSTWEIGHT, type=DATA);
15848 +   %em_getname(key=BOOSTSCORE,  type=DATA);
15849 +   %em_getName(key=BOOSTINFO,   type=DATA);
15851 +   %if &emloopnumber = 1 %then %do;
15852 +       filename gpDelta "&EM_FILE_CDELTA_TRAIN";
15853 +       data _null_;
15854 +          file gpdelta;
15855 +          put "if ROLE = 'TARGET' and NAME ne '&targetName' then delete;";
15856 +          put "else if upcase(NAME) eq '_RESAMP_' then do;";
15857 +          put "     LABEL='Arcing Frequency'; LEVEL='INTERVAL'; ROLE='FREQ';";
15858 +          put "end;";
15859 +       run;
15860 +       filename gpdelta;
15862 +       %let deleteString =;
15863 +       %if %sysfunc(exist(&em_user_boostweight)) %then %let deleteString = &deleteString &em_nodeid._boostweight;
15864 +       %if %sysfunc(exist(&em_user_boostscore))  %then %let deleteString = &deleteString &em_nodeid._boostscore;
15865 +       %if "&deleteString" ne "" %then %do;
15866 +           proc datasets lib=&em_lib nolist;
15867 +              delete &deleteString;
15868 +           run;
15869 +       %end;
15871 +       %emboost0( _train=&em_import_data, _dmboods=&em_user_boostweight, _target=&targetname, _freq=%em_freq);
15873 +       data &em_user_boostinfo;
15874 +          length _LOOP_ 8 _TARGET_ $32 _DMBOOST_DIVISOR_ 8 _SUMFREQ_ 8;
15875 +          _LOOP_            =&EM_PROPERTY_INDEXCOUNT;
15876 +          _TARGET_          ="&targetname";
15877 +          _DMBOOST_DIVISOR_ =&_dmboodiv;
15878 +          _SUMFREQ_         =&_sumfreq;
15879 +       run;
15880 +       data &em_user_emloopinfo;
15881 +          length MODE $8;
15882 +          label MODE     = "%sysfunc(sasmsg(sashelp.dmine, rpt_mode_vlabel,       NOQUOTE))"
15883 +                _LOOP_   = "%sysfunc(sasmsg(sashelp.dmine, rpt_groupIndex_vlabel, NOQUOTE))"
15884 +               _TARGET_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_target_vlabel,     NOQUOTE))";
15885 +           set &em_user_boostinfo(keep=_LOOP_ _TARGET_);
15886 +           MODE='Boosting';
15887 +       run;
15889 +       data &em_data_eminfo;
15890 +          length TARGET KEY $32  DATA $43;
15891 +          TARGET=''; KEY='BOOSTINFO'; DATA="&em_nodeid"; output;
15892 +       run;
15894 +   %end;
15895 +   %else %do;
15896 +       %global _dmboodiv;
15897 +       %let dmboosf = %upcase(&targetname);
15898 +       %if %sysfunc(length(&targetname))> 30 %then
15899 +           %let dmboosf =  %substr(&targetname, 1, 30);
15901 +       data _null_;
15902 +          set &em_user_boostinfo;
15903 +          call symput('_SUMFREQ',  strip(put(_sumfreq_,best12.)));
15904 +       run;
15906 +       %emboost2(_freq=%em_freq, _dmboods=&em_user_boostweight, _dmensds=&em_user_boostscore, _dmboosf=&dmboosf, _sumfreq=&_sumfreq);
15908 +       data &em_user_boostinfo;
15909 +          set &em_user_boostinfo;
15910 +          _DMBOOST_DIVISOR_ =&_dmboodiv;
15911 +       run;
15913 +       %emboost1(_tra=&em_export_train, _TRAIN=&em_import_data, _FREQ=%em_freq, _dmboods=&em_user_boostweight, _dmboodiv=&_dmboodiv);
15915 +   %end;
15917 +    %scoreValidTest(data=&em_import_validate, out=em_export_validate);
15918 +    %scoreValidTest(data=&em_import_test,     out=em_export_test);
15920 +   %doendbst:
15922 +%mend gp_boost;
15924 +%macro gp_group(numgroupvar=, groupvar=);
15925 +   %em_getname(key=GROUPINFO, type=DATA);
15926 +   %em_getName(key=LOOPINFO,  type=DATA);
15927 +   %em_getName(key=EMGROUPINFO, type=DATA);
15929 +   %if &emloopnumber=1 or ^%sysfunc(exist(&em_user_groupinfo)) %then %do;
15930 +       ods listing close;
15931 +       %if &numgroupvar = 1 %then %do;
15932 +           ods output onewayfreqs=temp;
15933 +       %end;
15934 +       %else %do;
15935 +           ods output crosstabfreqs=temp;
15936 +       %end;
15937 +       proc freq data=&em_import_data;
15938 +          table
15940 +          %let tableString =;
15941 +          %let freqString  =;
15942 +          %do i=1 %to &numgroupvar;
15943 +              %let tableString = &tableString %scan(&groupvar, &i, %str( ));
15944 +              %if &i<&numgroupvar %then %let tableString = &tableString *;
15945 +              %let freqString = &freqString.1;
15946 +          %end;
15947 +          &tableString / missing;
15948 +       run;
15949 +       ods listing;
15950 +       %if &numgroupvar = 1 %then %do;
15951 +           data _null_;
15952 +              retain totalobs 0;
15953 +              set temp end=eof;
15954 +              totalobs + frequency;
15955 +              if eof then
15956 +                 call symput('_totalObs_', put(totalobs, best.));
15957 +           run;
15959 +           proc sort data=temp out=loopinfo nodupkey;
15960 +              by &groupvar;
15961 +              where frequency>=&em_property_minimumObs;
15962 +           run;
15963 +       %end;
15964 +       %else %do;
15965 +           data _null_;
15966 +              set temp;
15967 +              where _type_='00';
15968 +              call symput('_totalObs_', put(frequency, best.));
15969 +           run;
15970 +           proc sort data=temp(drop=rowpercent colpercent) out=loopinfo nodupkey;
15971 +              by &groupvar;
15972 +              where _type_="&freqString" and frequency>=&em_property_minimumObs;
15973 +           run;
15974 +       %end;
15976 +       %em_getname(key=PRECODE, type=FILE, extension=sas);
15977 +       filename _pregrp "&em_user_precode";
15979 +       %let dsid = %sysfunc(open(loopinfo));
15980 +       %do i=1 %to &numgroupvar;
15981 +           %let varname = %scan(&groupvar, &i, %str( ));
15982 +           %let varnum  = %sysfunc(varnum(&dsid, &varname));
15983 +           %let varfmt  = %sysfunc(varfmt(&dsid, &varnum));
15984 +           %if %index(&varfmt, -) %then %do;
15985 +               %let varfmt = %sysfunc(scan(&varfmt, 1,-)).;
15986 +           %end;
15988 +           %let vartype = %sysfunc(vartype(&dsid, &varnum));
15989 +           %let varlen  = %sysfunc(varlen(&dsid, &varnum));
15990 +           %let fmttype&i = &vartype;
15991 +           %let fmtlen&i  = &varlen;
15993 +           %let fmtvar&i =;
15994 +           %let fmt&i    =;
15995 +           %if "&varfmt" ne "" %then %do;
15996 +              %let fmtvar&i  = _FORMAT&i._;
15997 +              %let fmt&i     = &varfmt;
15998 +           %end;
15999 +       %end;
16000 +       %let dsid = %sysfunc(close(&dsid));
16002 +       data _null_;
16003 +          %do i=1 %to &numgroupvar;
16004 +              %if "&&fmt&i" ne "" %then %do;
16005 +                  retain len_&i 0;
16006 +              %end;
16007 +          %end;
16009 +          set loopinfo end=eof;
16010 +          file _pregrp;
16012 +          %do i=1 %to &numgroupvar;
16013 +              %let varname = %scan(&groupvar, &i, %str( ));
16014 +              %if "&&fmt&i" ne "" %then %do;
16015 +                 if length(put(&varname, &&fmt&i)) > len_&i then
16016 +                    len_&i = length(put(&varname, &&fmt&i));
16017 +              %end;
16018 +          %end;
16020 +          if eof then do;
16021 +             %do i=1 %to &numgroupvar;
16022 +                 %let varname = %scan(&groupvar, &i, %str( ));
16023 +                 %if "&&fmt&i" ne "" %then %do;
16024 +                     put "length  &&fmtvar&i" len_&i "$" len_&i ";";
16025 +                     put "drop &&fmtvar&i" len_&i ";";
16026 +                     put "&&fmtvar&i" len_&i "= strip(put(&varname, &&fmt&i));";
16027 +                     call symput('fmtnewlen'!!"&i", put(len_&i, best.));
16028 +                     call symput('fmtnewvar'!!"&i", "&&fmtvar&i"!!strip(put(len_&i, best.)));
16029 +                 %end;
16030 +              %end;
16031 +          end;
16032 +      run;
16034 +      data loopinfo;
16035 +         set loopinfo;
16036 +         %do i=1 %to &numgroupvar;
16037 +             %let varname = %scan(&groupvar, &i, %str( ));
16038 +             %if "&&fmt&i" ne "" %then %do;
16039 +                 length  &&fmtnewvar&i $&&fmtnewlen&i;
16040 +                 &&fmtnewvar&i = strip(put(&varname, &&fmt&i));
16041 +             %end;
16042 +         %end;
16043 +      run;
16044 +      filename _pregrp;
16046 +      %if (&EM_PROPERTY_TARGETGROUP= Y and &EM_NUM_TARGET>1) %then %do;
16047 +          data temp;
16048 +             length _LOOP_ 8 _WHERE_  _WHEREDESC_ $2000;
16049 +             keep _LOOP_ _WHERE_ _WHEREDESC_ FREQUENCY;
16050 +      %end;
16051 +      %else %do;
16052 +          data &em_user_groupinfo;
16053 +             length _LOOP_ 8 _TARGET_ $32 _WHERE_  _WHEREDESC_ $2000;
16054 +             keep _LOOP_ _TARGET_  _WHERE_ _WHEREDESC_ FREQUENCY;
16055 +             retain _TARGET_ '';
16056 +      %end;
16058 +          set loopinfo;
16059 +          _LOOP_= _N_;
16060 +          _WHERE_ = '';
16061 +          _WHEREDESC_  = '';
16062 +          %do i=1 %to &numgroupvar;
16063 +              %let varname = %scan(&groupvar, &i, %str( ));
16064 +              %if "&&fmt&i" eq "" %then %do;
16065 +                  %if "&&fmtType&i" eq "N" %then %do;
16066 +                      _WHERE_     = strip(_WHERE_)!!" &Varname ="!!strip(put(&varname, best12.));
16067 +                      _WHEREDESC_ = strip(_WHEREDESC_)!!" &Varname ="!!strip(put(&varname, best12.));
16068 +                   %end;
16069 +                  %else %do;
16070 +                       _WHERE_ = strip(_WHERE_)!!" &Varname ='"!!tranwrd(strip(&varname), "'", "''")!!"'";
16071 +                       _WHEREDESC_ = strip(_WHEREDESC_)!!" &Varname ="!!strip(&varname);
16072 +                  %end;
16073 +              %end;
16074 +              %else %do;
16075 +                       _WHERE_ = strip(_WHERE_)!!" &&fmtnewvar&i ='"!!tranwrd(strip(&&fmtnewvar&i), "'", "''")!!"'";
16076 +                       _WHEREDESC_ = strip(_WHEREDESC_)!!" &VarName = "!!strip(&&fmtnewvar&i);
16077 +              %end;
16078 +              %if &i < &numgroupvar %then %do;
16079 +                  _WHERE_ = strip(_WHERE_)!!' and ';
16080 +                  _WHEREDESC_ = strip(_WHEREDESC_)!!' and ';
16081 +              %end;
16082 +          %end;
16083 +          %if "&EM_PROPERTY_MODE" = "CROSSVALIDATION"  %then %do;
16084 +              _WHERE_ = '^('!!strip(_WHERE_)!!')';
16085 +              _WHEREDESC_ = '^('!!strip(_WHEREDESC_)!!')';
16086 +              FREQUENCY = &_totalobs_ - FREQUENCY;
16087 +          %end;
16088 +      run;
16090 +      %if (&EM_PROPERTY_TARGETGROUP= Y and &EM_NUM_TARGET>1) %then %do;
16091 +          data target;
16092 +             length _TARGET_ $32;
16093 +              %do i=1 %to &em_num_target;
16094 +                  %let varname = %scan(%em_target, &i, %str( ));
16095 +                  _TARGET_ = "&varname";
16096 +                  output;
16097 +              %end;
16098 +          run;
16099 +          proc sql;
16100 +             create table &em_user_groupinfo as select * from  work.temp, work.target;
16101 +          quit;
16102 +          data &em_user_groupinfo;
16103 +             set &em_user_groupinfo;
16104 +             _LOOP_=_N_;
16105 +          run;
16106 +      %end;
16108 +      data &em_data_eminfo;
16109 +         length TARGET KEY $32  DATA $43;
16110 +         TARGET=''; KEY='GROUPINFO'; DATA="&em_nodeid"; output;
16111 +      run;
16113 +      data &em_user_emloopinfo;
16114 +         set &em_user_groupinfo;
16115 +         %if (&EM_PROPERTY_TARGETGROUP= Y and &EM_NUM_TARGET>1) %then %do;
16116 +             label _LOOP_      = "%sysfunc(sasmsg(sashelp.dmine, rpt_groupIndex_vlabel, NOQUOTE))"
16117 +                   _TARGET_    = "%sysfunc(sasmsg(sashelp.dmine, rpt_target_vlabel,     NOQUOTE))"
16118 +                   _WHEREDESC_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_where_vlabel,      NOQUOTE))"
16119 +                   FREQUENCY   = "%sysfunc(sasmsg(sashelp.dmine, rpt_count_vlabel,      NOQUOTE))";
16120 +             keep _LOOP_ _TARGET_ _WHEREDESC_ FREQUENCY;
16121 +         %end;
16122 +         %else %do;
16123 +             label _LOOP_      = "%sysfunc(sasmsg(sashelp.dmine, rpt_groupIndex_vlabel, NOQUOTE))"
16124 +                   _TARGET_    = "%sysfunc(sasmsg(sashelp.dmine, rpt_target_vlabel,     NOQUOTE))"
16125 +                   _WHEREDESC_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_where_vlabel,      NOQUOTE))"
16126 +                   FREQUENCY   = "%sysfunc(sasmsg(sashelp.dmine, rpt_count_vlabel,      NOQUOTE))";
16127 +             keep _LOOP_ _WHEREDESC_ FREQUENCY;
16128 +         %end;
16129 +      run;
16131 +      proc datasets lib=work nolist;
16132 +          delete temp loopinfo
16133 +          %if (&EM_PROPERTY_TARGETGROUP= Y and &EM_NUM_TARGET>1) %then %do;
16134 +              target
16135 +          %end;
16136 +          ;
16137 +      run;
16139 +   %end;
16141 +   filename gpDelta "&EM_FILE_CDELTA_TRAIN";
16142 +   %if (&EM_PROPERTY_TARGETGROUP= Y and &EM_NUM_TARGET>1) %then %do;
16143 +       data _null_;
16144 +          length quotedTarget $32;
16145 +          set &em_user_groupinfo;
16146 +          file gpdelta;
16147 +          if _N_ = &emloopnumber then do;
16148 +             quotedTarget = "'"!!strip(_TARGET_)!!"'";
16149 +             put "if ROLE = 'TARGET' and NAME ne " quotedTarget " then delete;";
16151 +             %do i=1 %to &numgroupvar;
16152 +                 %let gvar = %upcase(%scan(&groupvar, &i, %str( )));
16153 +                 put "if upcase(NAME)= '&gvar' then delete;";
16154 +             %end;
16155 +          end;
16156 +       run;
16157 +   %end;
16158 +   %else %do;
16159 +       data _null_;
16160 +          set &em_user_groupinfo;
16161 +          file gpdelta;
16162 +          if _N_ = &emloopnumber then do;
16163 +             %do i=1 %to &numgroupvar;
16164 +              %let gvar = %upcase(%scan(&groupvar, &i, %str( )));
16165 +                 put "if upcase(NAME)= '&gvar' then delete;";
16166 +             %end;
16167 +          end;
16168 +       run;
16169 +   %end;
16170 +   filename gpdelta;
16172 +   %let nobs=0;
16173 +   %let dsid = %sysfunc(open(&em_user_groupinfo));
16174 +   %if &dsid>0 %then %do;
16175 +       %let nobs = %sysfunc(attrn(&dsid, NOBS));
16176 +       %let dsid = %sysfunc(close(&dsid));
16177 +   %end;
16179 +   %if ^&nobs  %then
16180 +       %let emloopmode =;
16181 +   %else
16182 +       %if &nobs< &emloopnumber %then
16183 +           %let emloopmode = POSTLOOP;
16184 +        %else
16185 +        %let emloopmode = LOOP;
16186 +%mend gp_group;
16188 +%macro gp_Index;
16190 +   %em_getName(key=INDEXINFO,  type=DATA);
16191 +   %em_getName(key=EMLOOPINFO, type=DATA);
16193 +   %if &emloopnumber=1 or ^%sysfunc(exist(&em_user_indexinfo)) %then %do;
16194 +       data &em_user_indexinfo;
16195 +          length _LOOP_ 8 _TARGET_ $32 _WHERE_ $8;
16196 +          _LOOP_     =&EM_PROPERTY_INDEXCOUNT;
16197 +          _TARGET_   ="";
16198 +          _WHERE_    ="";
16199 +          output;
16200 +       run;
16201 +       data &em_data_eminfo;
16202 +          length TARGET KEY $32  DATA $43;
16203 +          TARGET=''; KEY='INDEXINFO'; DATA="&em_nodeid"; output;
16204 +       run;
16206 +       data &em_user_emloopinfo;
16207 +          length MODE $8;
16208 +          label MODE     = "%sysfunc(sasmsg(sashelp.dmine, rpt_mode_vlabel,       NOQUOTE))"
16209 +                _LOOP_   = "%sysfunc(sasmsg(sashelp.dmine, rpt_groupIndex_vlabel, NOQUOTE))"
16210 +          set &em_user_indexinfo(keep=_LOOP_ );
16211 +          MODE='Index';
16212 +       run;
16213 +   %end;
16215 +%mend gp_index;
16217 +%macro gp_Target;
16218 +   %let TargetName = %scan(%em_target, &emloopnumber);
16219 +   %if "&targetName" eq "" %then %do;
16220 +       %goto doendgpt;
16221 +   %end;
16223 +   %if %index(&EM_DEBUG, _ALL_) %then %do;
16224 +       %put GP_TARGET;
16225 +       %put &emLoopnumber &TargetName;
16226 +   %end;
16228 +   filename gpDelta "&EM_FILE_CDELTA_TRAIN";
16229 +   data _null_;
16230 +      file gpdelta;
16231 +      put "if ROLE = 'TARGET' and NAME ne '&targetName' then delete;";
16232 +   run;
16233 +   filename gpdelta;
16235 +   %if &emloopnumber eq 1 %then %do;
16236 +       %em_getName(key=TARGETINFO,  type=DATA);
16237 +       %em_getName(key=EMLOOPINFO,  type=DATA);
16239 +       data &em_user_targetinfo;
16240 +          length _LOOP_ 8 _TARGET_ $32 _WHERE_ $2000;
16241 +          retain _LOOP_;
16242 +          _WHERE_ = '';
16243 +          %do i=1 %to &em_num_target;
16244 +              %let varname = %scan(%em_target, &i, ' ');
16245 +              _TARGET_ = "&varname";
16246 +              _LOOP_+1;
16247 +              output;
16248 +           %end;
16249 +       run;
16250 +       data &em_user_emloopinfo;
16251 +         set &em_user_targetinfo(keep=_LOOP_ _TARGET_);
16252 +         label _LOOP_      = "%sysfunc(sasmsg(sashelp.dmine, rpt_groupIndex_vlabel, NOQUOTE))"
16253 +               _TARGET_    = "%sysfunc(sasmsg(sashelp.dmine, rpt_target_vlabel,     NOQUOTE))";
16254 +      run;
16256 +       data &em_data_eminfo;
16257 +          length TARGET KEY $32  DATA $43;
16258 +          TARGET=''; KEY='TARGETINFO'; DATA="&em_nodeid"; output;
16259 +       run;
16260 +   %end;
16262 +   %doendgpt:
16263 +%mend gp_target;
16265 +%macro train;
16266 +   %if %index(&EM_DEBUG, _ALL_) %then %do;
16267 +       %put LOOP MODE:   &emloopmode ;
16268 +       %put LOOP NUMBER: &emloopnumber;
16269 +   %end;
16271 +   %if &emloopmode eq 'POSTLOOP' %then %goto doendmain;
16273 +   %if ("&EM_PROPERTY_MODE" = "BOOSTING") or ("&EM_PROPERTY_MODE" = "BAGGING") %then %do;
16275 +       %let hpdmFlag = 0;
16276 +       %if %symexist(em_import_DATA_eminfo) %then %do;
16277 +            data _null_;
16278 +               set &em_import_DATA_eminfo;
16279 +               where KEY = "HPDMSAMPLE";
16280 +               call symput('hpdmFlag', '1');
16281 +            run;
16282 +       %end;
16283 +       %if &hpdmFlag %then %do;
16284 +           %let emexceptionstring = exception.server.EMTOOL.HPNOBOOSTBAG;
16285 +           %goto doendmain;
16286 +       %end;
16287 +   %end;
16289 +   %em_getname(key=EMLOOPINFO,  type=DATA);
16290 +   %em_getname(key=GROUPINFO,  type=DATA);
16291 +   %em_getname(key=BOOSTINFO,  type=DATA);
16292 +   %em_getname(key=BAGINFO,    type=DATA);
16293 +   %em_getname(key=TARGETINFO, type=DATA);
16294 +   %em_getname(key=INDEXINFO,  type=DATA);
16295 +   %em_getname(key=EMINFO,  type=DATA);
16297 +   %if "&emloopnumber" = "1" %then %do;
16298 +       %em_getname(key=PRECODE, type=FILE, extension=sas);
16299 +       filename _pregrp "&em_user_precode";
16300 +       data _null_;
16301 +         rc = fdelete('_pregrp');
16302 +       run;
16303 +       filename _pregrp;
16305 +       %let members =;
16306 +       %if %sysfunc(exist(&em_user_emloopinfo)) %then %let members = &members %scan(&EM_USER_EMLOOPINFO,   2, .);
16307 +       %if %sysfunc(exist(&em_user_groupinfo))  %then %let members = &members %scan(&EM_USER_GROUPINFO,  2, .);
16308 +       %if %sysfunc(exist(&em_user_targetinfo)) %then %let members = &members %scan(&EM_USER_TARGETINFO, 2, .);
16309 +       %if %sysfunc(exist(&em_user_boostinfo))  %then %let members = &members %scan(&EM_USER_BOOSTINFO,  2, .);
16310 +       %if %sysfunc(exist(&em_user_baginfo))    %then %let members = &members %scan(&EM_USER_BAGINFO,    2, .);
16311 +       %if %sysfunc(exist(&em_user_indexinfo))  %then %let members = &members %scan(&EM_USER_INDEXINFO,  2, .);
16312 +       %if %sysfunc(exist(&em_user_eminfo))     %then %let members = &members %scan(&EM_USER_EMINFO,  2, .);
16314 +       %if "&members" ne "" %then %do;
16315 +           proc datasets lib=&em_lib nolist;
16316 +              delete &members;
16317 +           run;
16318 +      %end;
16319 +   %end;
16321 +   %let emboostvar    =;
16322 +   %let embagvar      =;
16323 +   %let emgroupvar    =;
16324 +   %let emnumgroupvar =;
16325 +   %getVariables;
16327 +   %if &EM_PROPERTY_MODE = TARGET %then %do;
16328 +       %if &EM_NUM_TARGET>=1 %then %do;
16329 +           %gp_target;
16330 +           %if &emLoopNumber <= &EM_NUM_TARGET %then
16331 +               %let emloopmode= LOOP;
16332 +           %else
16333 +              %let emloopmode = POSTLOOP;
16334 +       %end;
16335 +       %else %do;
16336 +           %let emloopmode = ;
16337 +           %let emexceptionstring = exception.server.METADATA.USE1TARGET;
16338 +           %goto doendmain;
16339 +       %end;
16340 +   %end;
16341 +   %else
16342 +   %if &EM_PROPERTY_MODE = NOGROUP %then %do;
16343 +       %let emloopmode=;
16344 +       %if %sysfunc(exist(&em_data_eminfo)) %then %do;
16345 +           proc delete data=&em_data_eminfo;
16346 +           run;
16347 +       %end;
16348 +   %end;
16349 +   %else
16350 +   %if &EM_PROPERTY_MODE = INDEX %then %do;
16351 +       %gp_index;
16352 +       %if  &emLoopNumber <= &EM_PROPERTY_INDEXCOUNT %then
16353 +           %let emloopmode= LOOP;
16354 +       %else
16355 +           %let emloopmode= POSTLOOP;
16356 +   %end;
16357 +   %else
16358 +   %if "&EM_PROPERTY_MODE" = "STRATIFY" or "&EM_PROPERTY_MODE" = "CROSSVALIDATION"  %then %do;
16359 +       %if "&emloopnumber"="1" and "&emgroupvar" eq "" %then %do;
16360 +           %let emloopmode = ;
16361 +           %let emexceptionstring = exception.server.METADATA.USE1GROUPVAR;
16362 +           %goto doendmain;
16363 +       %end;
16365 +       %gp_group(numgroupvar=&emnumgroupvar, groupvar=&emgroupvar);
16366 +   %end;
16367 +   %else
16368 +   %if &EM_PROPERTY_MODE = BOOSTING %then %do;
16369 +       %if "&emboostvar" eq "" %then %do;
16370 +           %let emloopmode = ;
16371 +           %let emexceptionstring = exception.server.METADATA.USE1BOOSTVAR;
16372 +           %goto doendmain;
16373 +       %end;
16375 +       %if  &emLoopNumber <= &EM_PROPERTY_INDEXCOUNT %then %do;
16376 +           %let emloopmode= LOOP;
16377 +           %gp_boost(targetname=&emboostvar);
16379 +       %end;
16380 +       %else
16381 +           %let emloopmode= POSTLOOP;
16382 +   %end;
16383 +   %else
16384 +   %if &EM_PROPERTY_MODE = BAGGING %then %do;
16385 +       %if "&embagvar" eq "" %then %do;
16386 +           %let emloopmode = ;
16387 +           %let emexceptionstring = exception.server.METADATA.USE1BAGVAR;
16388 +           %goto doendmain;
16389 +       %end;
16391 +        %if  &emLoopNumber <= &EM_PROPERTY_INDEXCOUNT %then %do;
16392 +           %let emloopmode= LOOP;
16393 +           %gp_bag(targetname=&embagvar);
16394 +       %end;
16395 +       %else
16396 +           %let emloopmode= POSTLOOP;
16397 +   %end;
16400 +   %if &emloopmode eq "" %then %do;
16401 +       %let lib    = %scan(&em_data_eminfo, 1, .);
16402 +       %let member = %scan(&em_data_eminfo, 2, .);
16403 +       proc datasets lib=&lib nolist;
16404 +          delete &member;
16405 +       run;
16406 +   %end;
16408 +   %doendmain:
16410 +   %if %index(&EM_DEBUG, _ALL_) %then %do;
16411 +       %put NEW LOOP MODE: &emloopmode;
16412 +   %end;
16414 +   %if "&emloopmode" eq "POSTLOOP" %then %do;
16415 +       filename temp catalog 'sashelp.emutil.gp_postloop.source';
16416 +       %include temp;
16417 +       filename temp;
16419 +       %postloop;
16420 +   %end;
16422 +%mend train;
NOTE: %INCLUDE (level 1) ending.
NOTE: Fileref TEMP has been deassigned.
 
NOTE: DATA statement used (Total process time):
      real time           0.00 seconds
      cpu time            0.00 seconds
 
 
NOTE: Fileref _PREGRP has been deassigned.
 
NOTE: Deleting EMWS4.GRP_GROUPINFO (memtype=DATA).
NOTE: Deleting EMWS4.GRP_EMINFO (memtype=DATA).
 
NOTE: PROCEDURE DATASETS used (Total process time):
      real time           0.01 seconds
      cpu time            0.01 seconds
 
 
 
NOTE: There were 15 observations read from the data set EMWS4.GRP_VARIABLESET.
NOTE: DATA statement used (Total process time):
      real time           0.00 seconds
      cpu time            0.00 seconds
 
 
 
ERROR: The requested table is too large to process.
NOTE: There were 78792 observations read from the data set EMWS4.IDS_DATA.
NOTE: View EMWS4.VARSEL2_TRAIN.VIEW used (Total process time):
      real time           2.07 seconds
      cpu time            2.07 seconds
 
NOTE: There were 78792 observations read from the data set EMWS4.TRANS_TRAIN.
NOTE: The SAS System stopped processing this step because of errors.
NOTE: There were 78792 observations read from the data set EMWS4.VARSEL2_TRAIN.
NOTE: PROCEDURE FREQ used (Total process time):
      real time           2.10 seconds
      cpu time            2.10 seconds
 
WARNING: Output 'crosstabfreqs' was not created.  Make sure that the output object name, label, or path is spelled correctly.  Also, verify that the appropriate procedure options are used to produce the requested output object.  For example, verify that
         the NOPRINT option is not used.
 
 
ERROR: File WORK.TEMP.DATA does not exist.
 
NOTE: The SAS System stopped processing this step because of errors.
NOTE: DATA statement used (Total process time):
      real time           0.00 seconds
      cpu time            0.00 seconds
 
 
ERROR: File WORK.TEMP.DATA does not exist.
WARNING: No data sets qualify for WHERE processing.
 
NOTE: The SAS System stopped processing this step because of errors.
WARNING: The data set WORK.LOOPINFO may be incomplete.  When this step was stopped there were 0 observations and 0 variables.
NOTE: PROCEDURE SORT used (Total process time):
      real time           0.01 seconds
      cpu time            0.01 seconds
 
WARNING: Argument 2 to function VARFMT referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
WARNING: Argument 2 to function VARTYPE referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
WARNING: Argument 2 to function VARLEN referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
NOTE: Mathematical operations could not be performed during %SYSFUNC function execution. The result of the operations have been set to a missing value.
WARNING: Argument 2 to function VARFMT referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
WARNING: Argument 2 to function VARTYPE referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
WARNING: Argument 2 to function VARLEN referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
NOTE: Mathematical operations could not be performed during %SYSFUNC function execution. The result of the operations have been set to a missing value.
WARNING: Argument 2 to function VARFMT referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
WARNING: Argument 2 to function VARTYPE referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
WARNING: Argument 2 to function VARLEN referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
NOTE: Mathematical operations could not be performed during %SYSFUNC function execution. The result of the operations have been set to a missing value.
WARNING: Argument 2 to function VARFMT referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
WARNING: Argument 2 to function VARTYPE referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
WARNING: Argument 2 to function VARLEN referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
NOTE: Mathematical operations could not be performed during %SYSFUNC function execution. The result of the operations have been set to a missing value.
WARNING: Argument 2 to function VARFMT referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
WARNING: Argument 2 to function VARTYPE referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
WARNING: Argument 2 to function VARLEN referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
NOTE: Mathematical operations could not be performed during %SYSFUNC function execution. The result of the operations have been set to a missing value.
WARNING: Argument 2 to function VARFMT referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
WARNING: Argument 2 to function VARTYPE referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
WARNING: Argument 2 to function VARLEN referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
NOTE: Mathematical operations could not be performed during %SYSFUNC function execution. The result of the operations have been set to a missing value.
WARNING: Argument 2 to function VARFMT referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
WARNING: Argument 2 to function VARTYPE referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
WARNING: Argument 2 to function VARLEN referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
NOTE: Mathematical operations could not be performed during %SYSFUNC function execution. The result of the operations have been set to a missing value.
WARNING: Argument 2 to function VARFMT referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
WARNING: Argument 2 to function VARTYPE referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
WARNING: Argument 2 to function VARLEN referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
NOTE: Mathematical operations could not be performed during %SYSFUNC function execution. The result of the operations have been set to a missing value.
WARNING: Argument 2 to function VARFMT referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
WARNING: Argument 2 to function VARTYPE referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
WARNING: Argument 2 to function VARLEN referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
NOTE: Mathematical operations could not be performed during %SYSFUNC function execution. The result of the operations have been set to a missing value.
WARNING: Argument 2 to function VARFMT referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
WARNING: Argument 2 to function VARTYPE referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
WARNING: Argument 2 to function VARLEN referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
NOTE: Mathematical operations could not be performed during %SYSFUNC function execution. The result of the operations have been set to a missing value.
WARNING: Argument 2 to function VARFMT referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
WARNING: Argument 2 to function VARTYPE referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
WARNING: Argument 2 to function VARLEN referenced by the %SYSFUNC or %QSYSFUNC macro function is out of range.
NOTE: Mathematical operations could not be performed during %SYSFUNC function execution. The result of the operations have been set to a missing value.
 
 
 
 
NOTE: 0 records were written to the file _PREGRP.
NOTE: There were 0 observations read from the data set WORK.LOOPINFO.
NOTE: DATA statement used (Total process time):
      real time           0.01 seconds
      cpu time            0.01 seconds
 
 
 
NOTE: There were 0 observations read from the data set WORK.LOOPINFO.
NOTE: The data set WORK.LOOPINFO has 0 observations and 0 variables.
NOTE: DATA statement used (Total process time):
      real time           0.00 seconds
      cpu time            0.00 seconds
 
 
NOTE: Fileref _PREGRP has been deassigned.
NOTE: Line generated by the invoked macro "GP_GROUP".
358                                                                                          FREQUENCY = &_totalobs_ - FREQUENCY;
                                                                                                         -
                                                                                                         22
WARNING: Apparent symbolic reference _TOTALOBS_ not resolved.
 
ERROR 22-322: Syntax error, expecting one of the following: a name, a quoted string, a numeric constant, a datetime constant, a missing value, INPUT, PUT.
 
NOTE: Numeric values have been converted to character values at the places given by: (Line):(Column).
      1:1   1:1   1:1   1:1   1:1   1:1   1:1   1:1   1:1   1:1   1:1   1:1   1:1   1:1   1:1   1:1   1:1   1:1   1:1   1:1   1:1   1:1
NOTE: The SAS System stopped processing this step because of errors.
WARNING: The data set EMWS4.GRP_GROUPINFO may be incomplete.  When this step was stopped there were 0 observations and 5 variables.
NOTE: DATA statement used (Total process time):
      real time           0.01 seconds
      cpu time            0.01 seconds
 
 
 
NOTE: The data set EMWS4.GRP_EMINFO has 1 observations and 3 variables.
NOTE: DATA statement used (Total process time):
      real time           0.00 seconds
      cpu time            0.00 seconds
 
 
 
NOTE: There were 0 observations read from the data set EMWS4.GRP_GROUPINFO.
NOTE: The data set EMWS4.GRP_EMLOOPINFO has 0 observations and 3 variables.
NOTE: DATA statement used (Total process time):
      real time           0.03 seconds
      cpu time            0.03 seconds
 
 
 
NOTE: The file WORK.TEMP (memtype=DATA) was not found, but appears on a DELETE statement.
NOTE: Deleting WORK.LOOPINFO (memtype=DATA).
 
NOTE: PROCEDURE DATASETS used (Total process time):
      real time           0.00 seconds
      cpu time            0.00 seconds
 
 
 
 
NOTE: 0 records were written to the file GPDELTA.
NOTE: There were 0 observations read from the data set EMWS4.GRP_GROUPINFO.
NOTE: DATA statement used (Total process time):
      real time           0.00 seconds
      cpu time            0.00 seconds
 
 
NOTE: Fileref GPDELTA has been deassigned.
16424  *------------------------------------------------------------*;
16425  * End TRAIN: Grp;
16426  *------------------------------------------------------------*;
16427
 
*------------------------------------------------------------*
*
* ERROR: Run time error was encountered.  The system error returned was 3000.
* Please report unresolved problems to Technical Support.
*
*------------------------------------------------------------*

Video on the unsupervised learning technique, autoencoders

AnnaBrown — Tue, 03 Mar 2020 20:00:43 GMT

Hello there,

Check out 's latest video explaining autoencoders. He takes a look at two types of autoencoders -- denoising and sparse -- and explains how to train them in SAS. Have a look:

(Comments are closed on this message -- but visit YouTube and leave a comment on the video. Subscribe to the SAS Users YouTube channel to get more like it!)

Enterprise Miner - Decision Tree - Cross Validation

JKarp_11 — Tue, 03 Mar 2020 10:46:06 GMT

Dear community,

I need to better understand what the property „Perform Cross Validation“ in the section „Cross Validation“ for a decision tree does in general.

For me the purpose of cross validation (CV) is not to help select a particular tree (as the final model) but rather to qualify a model (which is created by 100% of the training sample before the CV), i.e. to provide metrics such as the average MSE (average of all “sub-trees” generated by the CV) which can be useful in asserting the level of precision one can expect from the application.

Now I have run two trees separately, one with “Perform Cross Validation”=yes and one without. The trees are different, i.e. the tree with CV=yes has less leaves. According to this outcome I assume that the enterprise miner uses a specific tree created by the CV as the final model (probably the one with the smallest MSE). I.e. a tree which is trained by 100-X% instead of 100% of the initial training sample.

Or does the results of the cross validation (average MSE) are used for pruning the original tree? However in this case pruning would be executed after CV…In my case I have selected the pruning property method “assessment” in section subtree.

I already thank you for your precious assistance! As it is a general question I hope this can be answered without data, codes.

Best regards

Hyperparameter Tuning in SAS Enterprise Miner

RobWobDobBlobb — Tue, 25 Feb 2020 10:30:33 GMT

Hi all,

I've been wondering if there is a way to do hyperparameter tuning for your model in SAS Enterprise Miner.

Doing it manually is no option and using a Start-End node I only manage to do things like k-fold cross validation but not actual hyperparameter tuning.

I've come across it in SAS Viya, e.g. in this paper: Automated Hyperparameter Tuning for Effective Machine Learning

Or interfaces bridging to R for ml(R): Ein SAS Enterprise Miner Interface für systematisches Hyperparameter-Tuning mit (ml)R

As on our servers there is no SAS Viya, no SAS VDMML, no R, etc. but only SAS EG and SAS EM, so a solution within SAS EM (14.3) would be great.

I'm happy for any input, cheers

SAS Viya 3.5 Model Studio pipeline result libraries

sunmi — Mon, 10 Feb 2020 08:16:20 GMT

While I get used to handling Model Studio on SAS Viya 3.5, I have gotten a question about how to connect pipeline libraries in Model Studio. As an example, I attached the captured result of stepwise logistic regression with HMEQ data. There are also download buttons that help to get result files as an excel, which means result data sets are saved somewhere on CAS.

The point is, I want to connect CAS libraries which are derived when I execute pipelines and also want to check data set as ‘.sasbdat’ format in SAS Studio.

I already tried to read logs and run several codes like below, but it never worked!

(* sample logs from logistic regression node that I refer to.)

I'm looking forward who has some solution..

Thanks!

Video: Gradient boosting explained

AnnaBrown — Thu, 20 Feb 2020 16:10:59 GMT

In this tutorial video, let SAS’ Christa Cody give you a high-level introduction to gradient boosting (i.e., no complicated math involved). Then, she builds a machine learning pipeline in SAS Model Studio containing a Gradient Boosting node and explores its options.

Want more? Here are some related community articles on gradient boosting:

Asked & Answered: How can I pass data from an Open Source Code node to next node in SAS Model Studio

Quick tips for setting your Gradient Boosting node properties in SAS® Enterprise Miner™

Assessing Models by using k-fold Cross Validation in SAS® Enterprise Miner™

(Comments are closed on this message -- but visit YouTube and leave a comment on the video. Subscribe to the SAS Users YouTube channel to get more like it!)

Enterprise Miner is not recognizing variables denoted as target

miwilliams — Wed, 19 Feb 2020 20:56:39 GMT

Hi all,

I feel like I am missing some sort of huge "aha" on this, but I simply don't know enough to help myself. I am trying to run various models in SAS EM on a data set of 5 different variables (age ranges, sex, number of recipients, geographic region, and diagnosis). This is the first time I have imported the data, which is an Excel spreadsheet. All of the setup seemed to go through fine; it appears all lines (just under 1700) were imported, and there are no missing data.

However, from the time I tried running decision tree, interactive decision tree, regression, and even an auto neural, the run fails, indicating I need at least one target variable. I have tried designating each variable as such, but it is still not recognizing as intended, and I am at a loss.

I hope the screen shots below will help assist in seeing what I have done. I appreciate any help in the mean time. Thanks in advance!

Event Based Sampling in Model Studio SAS Viya

andreas_zaras — Tue, 18 Feb 2020 21:53:20 GMT

Hello!

I am testing Model Studio DM and ML using the pva97nk data set. This data set is oversampled so the proportion of 1's and 0's has become 50% - 50% from an initial proportion of 5% - 95% (1's - 0's again).

In SAS EM in order to adjust for balanced sampling we had to put 0.05 - 0.95 in the prior probabilities tab of the decision property. Now in SAS Model Studio of SAS Viya i go to the project settings and i put Event=5 Non-Event=95 in the Event Based Sampling Tab.

When i run the pipeline it stops in the data node and the following message is output in the log:

NOTE: Oversampling is activated.
NOTE: Using SEED=12345 for sampling.
ERROR: There is not enough observations from non-event level to satisfy the event proportion.
ERROR: The action stopped due to errors.

What is wrong with the above process?

Thanks in advance,

Andreas

Please Help: Score Node Run Time Error in Enterprise Miner

ECUHart — Tue, 18 Feb 2020 19:20:57 GMT

Hi,

I need help. I'm working on an assignment for my Data Mining course and my score node won't function properly. It keeps pulling a run time error:

treeconversion=0

14286 %let syscc=0;

14287 filename _wipchk catalog "EMWS1.CNTRL.test.source";

14288 data _null_;

14289 file _wipchk;

14290 put '/* Test */';

14291 run;

NOTE: The file _WIPCHK is:

Catalog Name=EMWS1.CNTRL.TEST.SOURCE,

Catalog Page Size=4096,

Number of Catalog Pages=5,

Created=Mon, Feb 17, 2020 09:03:39 AM,

Last Modified=Tue, Feb 18, 2020 01:45:51 PM,

Filename=U:\House Prices\House Prices\Workspaces\EMWS1\cntrl.sas7bcat,

Release Created=9.0401M1,Host Created=X64_7PRO

NOTE: 1 record was written to the file _WIPCHK.

The minimum record length was 10.

The maximum record length was 10.

NOTE: DATA statement used (Total process time):

real time 0.01 seconds

cpu time 0.01 seconds

14292 data _null_;

14293 rc = fdelete('_wipchk');

14294 run;

NOTE: DATA statement used (Total process time):

real time 0.01 seconds

cpu time 0.00 seconds

14295 filename _wipchk;

NOTE: Fileref _WIPCHK has been deassigned.

14296 filename _wipxml 'C:\Users\hartb10\AppData\Local\Temp\45\SAS Temporary Files\_TD25404_VCLCTX-XA-03_\Prc2\DiagramOpenSessionResponse.xml' encoding="UTF-8" NOBOM;

34 The SAS System 13:23 Tuesday, February 18, 202

WARNING: End of file.

Can anyone help?

Thanks,

QTMS optimal binning macros

DocMartin — Thu, 13 Feb 2020 18:51:39 GMT

Does anyone know where I can find information on the set of macros called QTMS, which is supposed to generate optimal bins given a target? Thanks! Andrew

How does EM calculate difference scores in incremental node

hsawhney — Tue, 11 Feb 2020 03:15:54 GMT

Hello all, I am confused how exactly SAS EM calculates difference scores for each observation in netlift modeling using incremental model node. My guess is that it scores the same observation for Treatment = 1 and Treatment = 0 using the same model and taking difference of both probabilities. Is that understanding correct? If not, how are the difference scores calculated?

Enterprise miner error msg when running HP Tree

CathyGu — Tue, 28 Jan 2020 19:14:46 GMT

when I was running the HP tree for a regression problem. The following error occurred with the error message of insufficient resources to proceed. Data size is about 156M records, with 70/30 split for train/validation. Please help. Thanks.

NOTE: External file /model/conr_out/qzrzct/New Folder/creditlossprediction/Workspaces/EMWS5/HPTree/EMFLOWSCORE.sas opened.

NOTE: External file /model/conr_out/qzrzct/New Folder/creditlossprediction/Workspaces/EMWS5/HPTree/ENGLISHRULES.txt opened.

NOTE: The HPSPLIT procedure is executing in single-machine mode.

ERROR: Insufficient resources to proceed.

NOTE: View WORK.HPTREE_TRAINDATA.VIEW used (Total process time):

real time 27:23.02

cpu time 26:19.49

NOTE: There were 111695660 observations read from the data set EMWS5.HPPART_TRAIN.

NOTE: There were 47869558 observations read from the data set EMWS5.HPPART_VALIDATE.

NOTE: The SAS System stopped processing this step because of errors.

NOTE: There were 159565218 observations read from the data set WORK.HPTREE_TRAINDATA.

WARNING: The data set EMWS5.HPTREE_VARIMP may be incomplete. When this step was stopped there were 0 observations and 0 variables.

WARNING: Data set EMWS5.HPTREE_VARIMP was not replaced because this step was stopped.

WARNING: The data set EMWS5.HPTREE_OUTSUBTREE may be incomplete. When this step was stopped there were 0 observations and 0 variables.

WARNING: Data set EMWS5.HPTREE_OUTSUBTREE was not replaced because this step was stopped.

WARNING: The data set EMWS5.HPTREE_TREEPLOT may be incomplete. When this step was stopped there were 0 observations and 0 variables.

WARNING: Data set EMWS5.HPTREE_TREEPLOT was not replaced because this step was stopped.

WARNING: The data set EMWS5.HPTREE_OUTPRUNESTATS may be incomplete. When this step was stopped there were 0 observations and 0 variables.

WARNING: Data set EMWS5.HPTREE_OUTPRUNESTATS was not replaced because this step was stopped.

NOTE: PROCEDURE HPSPLIT used (Total process time):

real time 27:27.50

cpu time 26:23.76

SAS Enterprise Miner Text Mining problems

PackaLacken — Mon, 27 Jan 2020 03:07:34 GMT

I am working on a Case study out of one of my textbooks for my Master's in Data Analytics programs and am having some problems with SAS Enterprise Miner. My data sets are comprised of two columns, name and text. Name is the name of the file the text comes from, and text is the text from that file.

1. I have 4 data sets ( sas1976_1980, sas1981_1990, sas1991_2000, and sas2001_2012) trying to add two of the four (sas1981_1990, sas1991_2000)as a data source I am presented with this message

2. With my other two data sets that can be added in fine I get this error when trying to run a Text Parsing node on it

Any help is greatly appreciated and I have included the data sets as attachments!

RUN TIME ERROR ON EM WHEN RUNNING StatExplore NODE DUE TO MEMSIZE

emmaadiosyahoo — Sat, 18 Jan 2020 12:17:10 GMT

Please the following error was encountered when running StatExplore node on a data with 10 variables including the target variable in SAS MINER via SAS ON DEMAND FOR ACADEMICS. Urgent help needed so that I can complete my modelling.

ERROR: PROC ARBORETUM needs at least 8258396524 bytes of RAM memory for this problem. Only 3408015360 bytes available.
ERROR: Ignoring ASSESS statement.
11534 SUBTREE BEST;
WARNING: SUBTREE statement ignored. Type RUN; to continue running the procedure or QUIT; to stop.
11535 save RULES=WORK.Stat_RULE;
WARNING: SAVE statement ignored. Type RUN; to continue running the procedure or QUIT; to stop.
11536 run;
11537 quit;
WARNING: The data set WORK.STAT_RULE may be incomplete. When this step was stopped there were 0 observations and 0 variables.
*------------------------------------------------------------*
*
* ERROR: Run time error was encountered. The system error returned was 1012.
* Please report unresolved problems to Technical Support.
*

Factorization Machines in VDMML Viya

andreas_zaras — Mon, 13 Jan 2020 14:26:01 GMT

Hello!

I ran a factorization machine model in a data set and i got predictions for the 500000 non missinh ratings. I have exported the ASTORE table. I read that i ahve to follow the below process to get predictions for the missing ratings in the data set in SAS studio:

options cashost="server" casport=5570;

cas mysess;

cas;

caslib _all_ assign;

proc casutil;

load casdata=" model.sashdat " incaslib="Models" casout="test" outcaslib=casuser;

quit;

proc astore;

score data= public.Final_Nwtflix_DataSet out=casuser.scored rstore=casuser.test;

run;

What options should i put for the server and the port since i am using a trial version of VDMML - Viya?

Thanks

Multi-label Classification in SAS Viya

NichelleT — Fri, 31 Jan 2020 03:13:38 GMT

Hi everyone,

What is the best way in SAS to solve a Multi-label classification Problem?

Description of data: There are 10 binary explanatory variables and would like to predict 5 binary target variables at the same time.

We have tried to use SAS Model studio but only one target variable can be assigned for one data set.

I have done some research online and found that in Enterprise Miner there can be multiple target variables, can we achieve the same outcome in SAS Viya as Enterprise Miner is not available in my company.

Any suggestions in using SAS Model studio or SAS Studio are welcomed.

Thank You.

SAS Viya Connection failed and Error invoking action

paulaxa1 — Sat, 11 Jan 2020 14:26:23 GMT

I am trying to run autotune on some datasets for evaluation, but for some reason, on one small dataset it runs fine, but on a larger one, I get a bunch of errors. I run the online version of SAS Viya, I haven't downloaded anything.

The errors I get with the bigger dataset are "Error invoking action" and "Connection failed. Server returned: SAS Logon Manager authentication failed: Access denied"

Any help would be appreciated.

Thanks in advance.

cas mysess sessopts=(nworkers=1);

libname mycaslib cas casref=mysess;

filename myfldr filesrvc folderPath = '/Public/Projects/';
%include myfldr ('SAS2017-0514_benchmark_datasets.sas'); /*source2;*/


/* Tune a Decision Tree to each benchmark problem */
/* -----------------------------Diabetes -------------------------------- */
proc cas noqueue;
    print "---TWONORM / TUNE DECISION TREE ---";
    autotune.tuneDecisionTree result=r /
        trainOptions={
            table={name='Twonorm', vars={{name='X1'}, {name='X2'}, {name='X3'}, {name='X4'}, {name='X5'}, {name='X6'}, {name='X7'}, {name='X8'}, {name='X9'}, {name='X10'}, 
                                                {name='X11'}, {name='X12'}, {name='X13'}, {name='X14'}, {name='X15'}, {name='X16'}, {name='X17'}, {name='X18'}, {name='X19'}, {name='X20'},{name="Y"}}}, 
			inputs={{name='X1'}, {name='X2'}, {name='X3'}, {name='X4'}, {name='X5'}, {name='X6'}, {name='X7'}, {name='X8'}, {name='X9'}, {name='X10'}, 
                                                {name='X11'}, {name='X12'}, {name='X13'}, {name='X14'}, {name='X15'}, {name='X16'}, {name='X17'}, {name='X18'}, {name='X19'}, {name='X20'}},
            target='Y', 
            nominals={'Y'}, 
            casOut={name='dt_twonorm_model', replace=true}, 
            nbins=20, maxlevel=11, crit='GAINRATIO', maxbranch=2, leafsize=5, 
            missing='USEINSEARCH', minuseinsearch=1, 
            binorder=true, varimp=true, mergebin=true, encodeName=true
        }
	tunerOptions={nsubsessionworkers=1, nparallel=10, popsize=31, maxTime=300}
    ;
    print r;
run;
quit;

ERROR: Session option NWORKERS can only be set at session startup.

ERROR: The action stopped due to errors.

ERROR: Connection failed. Server returned: SAS Logon Manager authentication failed: Access denied.

ERROR: Error invoking action 'dtreeTrain'.

ERROR: Error invoking action 'dtreeTrain'.

ERROR: Error invoking action 'dtreeTrain'.

ERROR: Error invoking action 'dtreeTrain'.

ERROR: Error invoking action 'dtreeTrain'.

ERROR: Insufficient resources to perform the analytic operation.

ERROR: The action stopped due to errors.

ERROR: Connection failed. Server returned: SAS Logon Manager authentication failed: Access denied.

ERROR: Error invoking action 'dtreeTrain'.

ERROR: Error in the tuneDecisionTree action.

ERROR: The action stopped due to errors.

WARNING: Limit set by ERRORS= option reached. Further errors of this type will not be printed.

WARNING: Limit set by ERRORS= option reached. Further errors of this type will not be printed.

WARNING: Limit set by ERRORS= option reached. Further errors of this type will not be printed.

WARNING: Limit set by ERRORS= option reached. Further errors of this type will not be printed.

WARNING: Limit set by ERRORS= option reached. Further errors of this type will not be printed.

WARNING: Objective evaluation 2 was terminated.

NOTE: The CAS statement request to update one or more session options for session MYSESS completed.

NOTE: Libref MYCASLIB was successfully assigned as follows:

NOTE: The infile IN is:

NOTE: Invalid data for Y in line 1 1-4.

NOTE: Invalid data for X1 in line 1 6-9.

NOTE: Invalid data for X2 in line 1 11-17.

NOTE: 5301 records were read from the infile IN.

NOTE: The data set MYCASLIB.BANANA has 5301 observations and 3 variables.

NOTE: DATA statement used (Total process time):

NOTE: The infile IN is:

NOTE: Invalid data for X1 in line 1 7-17.

NOTE: Invalid data for X2 in line 1 19-30.



NOTE: 5002 records were read from the infile IN.

NOTE: SAS went to a new line when INPUT statement reached past the end of a line.

NOTE: The data set MYCASLIB.WAVEFORM has 5001 observations and 41 variables.

NOTE: DATA statement used (Total process time):

NOTE: Active Session now MYSESS.

NOTE: Autotune is started for 'Decision Tree' model.

NOTE: Autotune option SEARCHMETHOD='GA'.

NOTE: Autotune option MAXTIME=300 (sec.).

NOTE: Autotune option SEED=458531802.

NOTE: Autotune objective is 'Misclassification Error Percentage'.

NOTE: Autotune number of parallel evaluations is set to 10, each using 0 worker nodes.

NOTE: Added action set 'decisionTree'.

NOTE: Added action set 'decisionTree'.

NOTE: Action 'dtreeTrain' failed due to insufficient resources.

NOTE: PROCEDURE CAS used (Total process time):





1     OPTIONS NONOTES NOSTIMER NOSOURCE NOSYNTAXCHECK;
75    
76    cas mysess sessopts=(nworkers=2);

How to use infile on an uploaded CSV file

paulaxa1 — Fri, 10 Jan 2020 18:49:51 GMT

I am using a demo version of SAS Viya. I want to recreate the benchmark experiments for of the hyperparameter optimization tool "autotune". The version I use is online, I haven't downloaded anything, I just log in and I have access to SAS Studio. I have uploaded the CSV files I want to use, they are under the "Explorer", in the SAS Content/Users/(my username)/dataCSV folder (See Image).

When I right click on the file and copy the file location, and use that file location for the infile statement, I get the "ERROR: Physical file does not exist" error.

This is a sample of the code, same error same problem. What should I do? (again I am using online version I haven't downloaded anything). Thanks in advance.

data work.Banana;
   infile "/Users/paulxaxaxaxa@gmail.com/dataCSV/banana.csv" delimiter=',';  /* set path appropriately */
   input 
      Y
      X1
      X2 ;
run;

What real world want from Data Scientist

sdhilip — Fri, 10 Jan 2020 01:56:27 GMT

There is a wide gap between University and real-world in regards to Data Science. Non-technical skills are equally important to become a Data Scientist. Below skills plays a major role in order to work as a Data Scientist. For more details, please check this link

✔️ Understanding the business problem

✔️ Teamwork

✔️ Being A Good Listener

✔️ Documentation

✔️ Agile environment

✔️ Storytelling

✔️ Creativity in showing the output

✔️ Ask for help

✔️ Passion

✔️ Keep Learning!

✔️ Using Version Control

✔️ Coding

Thanks for the read. I am writing more beginner-friendly posts SAS Community. Follow me up at SAS Community.

Supervised Machine Learning Procedures Using SAS Viya in SAS Studio

sasmlp — Fri, 03 Jan 2020 21:20:05 GMT

Do you want to learn more about the data science methods in SAS Viya? The Supervised Machine Learning Procedures Using SAS® Viya® in SAS® Studio course covers predictive modeling techniques such as linear and logistic regression, decision tree and ensemble of trees (forest and gradient boosting), neural networks, support vector machines, and factorization machines. Come join us on January 28-31 on live web or March 19-20 in the New York Training Center. Register using the link below:

https://support.sas.com/edu/schedules.html?ctry=us&crs=DMML

2 video tutorials on dropout and batch normalization in deep learning

AnnaBrown — Fri, 03 Jan 2020 14:44:01 GMT

Hi there community, Happy New Year!

Here are a couple of deep learning-related tutorials to get you going.

I think you’ll enjoy the explanation of dropout in deep learning by in this tutorial. He tells a story about a “nose neuron” in a training process. “Dropout forces neurons in your model to become more generalists as opposed to specialists,” he explains.

then tells us how to use batch normalization in a deep learning model. Batch normalization is typically used to solve – or at least mitigate – the internal covariate shift problem. Watch to learn more.

(Comments are closed on this message -- but visit YouTube and leave a comment on the video. Subscribe to the SAS Users YouTube channel to get more like it!)

Anna

Enterprise Miner Decision Tree

ZaneleD — Thu, 02 Jan 2020 11:36:50 GMT

Its my first time building a model. So I am using SAS videos to build the model. I am using one data source . I partitioned it into 60% training, 30% Validation and 10% testing. Now I got an error when running a decision tree.

log Error: Must specify a training or raw data set.;

May you kindly assist:

Thank you.

screenshot of the process flow:

SAS Enterprise Miner 14.1 Decision Tree node: Unstable Results

yunfeizhao100 — Fri, 27 Dec 2019 07:11:53 GMT

We are using SAS Enterprise Miner 14.1. I have the below question for SAS Enterprise Miner. I am not sure where to get help, but I want to start from here. If you need more information, please let me know!

These questions are about to build a 0/1 binary target decision tree, where 1 is < 5% in data.

***Question 1, I open Interactive Tree window, I click Split Node, and only see 3-5 input attributes listed for use, and I am expecting nearly 100 input attributes? How can I get all my input attributes?

***Question 2, I run a Decision Tree node, and then I open Interactive Tree window. I see a few layers and nodes on the tree. I prune off part of the tree, and I want to Split Node from where I did the Prune Node. But I only see 3-5 input attributes, and I am expecting more input attributes. I prune off all nodes, and only keep the root node, then I save, close, and re-open the Interactive Tree, I might see more input attributes. What is happening here? What should I do?

***Question 3, My Decision Tree is typical 0/1, bad or not bad. But the Splitting Rule does not have Binary Target Criterion? What does it use for Binary Target? Interval Target Criterion, Nominal, or Ordinal?

***Question 4, I have millions of records, and nearly 100 input attributes. I basically use default values for Decsion Tree node, but change Use Input Once to Yes, change Maximum Branch to 4, leave maximum depth at 6, and click RUN, and it produces a tree. This tree gives me many leaves that only have a handful of records which is no use at all. So I change Leaf Size to ~2% of total records, now it produce a tree with only a single root node? What is happening here? What should I do?

***Question 5, What are typical property settings for Decision Tree node to build a 0/1 binary target tree (where 1 < 5% in the records), and to build a interval target tree, say outstanding balance? I have years of experience in using another commercial tool where I use default setting for everything, and results are as expected. I am fairly new to Eminer for the Decision Tree node and Data Source node, and I want to get started from expert advice here.

SAS Enterprise Miner 14.1 Decision Tree node: how to use it in credit card strategy

yunfeizhao100 — Fri, 27 Dec 2019 06:57:11 GMT

***Question 1, I am building a simple 0/1 binary target decision tree, where 1 is credit default which is a rare event < 5% in data. I do not look at profit or validation or other decision. I simply want to see what input attributes can best split default customers vs non-default customers. How should I set up the property of Decision Tree node? I get confused by a long list of Decision Tree node properties, including subtree property.

***Question 2, I am building a decision tree, where the target is the outstanding balance for a credit card account. I do not look at profit or validation or other decision. I simply want to see what input attributes can best split balance into different levels. How should I set up the property of Decision Tree node? I get confused by a long list of Decision Tree node properties, including subtree property.

SAS Enterprise Miner 14.1 Decision Tree node is not using all records in training

yunfeizhao100 — Fri, 27 Dec 2019 06:22:41 GMT

I am simply connecting a Data Source node to a Decision Tree node. When I create the data source, I set it up to not to create a sample and the data role is RAW. When I set up Decision Tree node, I use everything by default, except that for Interactive part, I set up by None for sample part. I click RUN, the results show that only a sample was used for the Decision Tree node (see the log below, 2328740 out of 3195296 training cases are shown in the root node). Then I open the Interactive Tree, it shows the same, only part of the records are used in the root node, not all the records. I am not sure why only part of the records are being used for the Decision Tree node. Is this because of some sort of memory limitation, or the Data Source or Decision Tree node property settings that I am not aware of? Thanks you for sharing your thoughts.

NOTE: Deleting WORK.CLASSOUT (memtype=DATA).
NOTE: Deleting WORK.VAROUT (memtype=DATA).

NOTE: PROCEDURE DATASETS used (Total process time):
real time 0.00 seconds
user cpu time 0.00 seconds
system cpu time 0.00 seconds
memory 163511.14k
OS Memory 173656.00k
Timestamp 12/26/2019 04:08:59 PM
Step Count 1 Switch Count 0
Page Faults 0
Page Reclaims 6
Page Swaps 0
Voluntary Context Switches 0
Involuntary Context Switches 0
Block Input Operations 0
Block Output Operations 8

NOTE: Data file EZHAO.EZHAO_MAST_EC.DATA is in a format that is native to another host, or the file encoding does not match the session encoding. Cross Environment Data Access will be used, which might require additional CPU resources and might reduce
performance.
NOTE: 3220450 kilobytes of physical memory.
NOTE: Will use 2328740 out of 3195296 training cases.
NOTE: Using memory pool with 1020312576 bytes.