All SAS Data Mining and Machine Learning posts

Re: Solution for imbalanced data, categorical target, 98% weights on 0 outcomes.

Ksharp — Tue, 11 Aug 2020 11:54:18 GMT

Due to every small event probability , any model would not be trusted.

Oversample stands for enhancing event prob, if you have 1000 obs only 10 obs is 1,you need randomly sample 30 or 40 from the remain 990 obs which is 0 to form a train data to model . a.k.a 1:0 is about 1:3 or 1:4 .

if you are using PROC LOGISTIC ,don't forget to use PEVENT=0.01 to adjust predicted prob .

And maybe have good ideas.

Re: Solution for imbalanced data, categorical target, 98% weights on 0 outcomes.

janex — Tue, 11 Aug 2020 03:56:18 GMT

Could you please take a look at the question of mine that I posted in my page ? Thank you very much.

Re: Solution for imbalanced data, categorical target, 98% weights on 0 outcomes.

akmsharif — Mon, 10 Aug 2020 21:12:33 GMT

Can you please let me know a bit detail?

In Pipeline comparision tab, what is the use of holdout data set

kbharat439 — Mon, 10 Aug 2020 17:55:20 GMT

Please explain the use of score a holdout data set.

Re: Change the champion model

kbharat439 — Mon, 10 Aug 2020 17:43:26 GMT

Thanks for the solution. It helped me a lot.

Re: Change the champion model

BrettWujek — Mon, 10 Aug 2020 17:17:30 GMT

You can change the champion model by selecting it in the table and using the action menu in the upper right as shown in the screenshot below.

Re: Solution for imbalanced data, categorical target, 98% weights on 0 outcomes.

Ksharp — Mon, 10 Aug 2020 11:24:05 GMT

Oversample , make good:bad about 3:1 or 4:1 .

Solution for imbalanced data, categorical target, 98% weights on 0 outcomes.

akmsharif — Mon, 10 Aug 2020 06:14:36 GMT

I am trying to do logistic regression, decision tree, KNN & neural network on a dataset where I have 9800 rows, the target is binary and 98% 0. I have 1000 interval predictors, all the variables have many 0s and not normal distributions. How should I approach to handle the imbalanced data in SAS Miner for each of the models? Can somebody pls help?

Re: Binning and Pre-Binning in Interactive Grouping

ronya — Sun, 09 Aug 2020 10:46:24 GMT

Many thanks for the detailed response and recommendation. I've had a chance to run the 2-stage process and see the binning/grouping process and their coarse/fine views

Re: Binning and Pre-Binning in Interactive Grouping

WendyCzika — Fri, 07 Aug 2020 15:12:13 GMT

Yes, that is all correct!

Change the champion model

kbharat439 — Thu, 06 Aug 2020 11:22:47 GMT

Please let me know the process of changing a champion model in the pipeline comparision tab.

Re: Binning and Pre-Binning in Interactive Grouping

pvareschi — Wed, 05 Aug 2020 14:42:58 GMT

Someone from SAS may be able to provide a more accurate respose, but, as far as I know, the algorithm behind the Interactive Grouping Node uses a two-step approach for interval variables:

1. First, it "discretizes" the variables by creating groups, essentially transforming the variables from interval to nominal

2. Secondly, applies a Tree-based logic to find the optimal binning based on the groups from step (1)

My understanding is the above approach is used only for computational efficiency reasons, because, in general, interval variables may have hundreds, if not, thousands of different values whihc would make it too computational intensive for a Tree algorithm to fully evaluate.

Therefore, by carrying out a pre-binning step, you end up with far fewer categories which then can be optimised based on a Tree-like algorithm.

Lastly, from my experience, unless you have a good reason for using "bucket", my advice is to always go for "quantile" (i.e. that should be the default approach unless, for some specific reason, you want to have groups defined by having the same width).

Re: Binning and Pre-Binning in Interactive Grouping

Ksharp — Wed, 05 Aug 2020 10:57:57 GMT

I think quantile, bucket and tree are just three bin methods , you can use one of them .
Someone more like Tree , Someone more like quantile.

You could bin many groups like 20 by quantile, bucket method, and merge any two groups into one group to make Chisquare or Gini max , and so on , I think that is a tree method.

Re: Binning and Pre-Binning in Interactive Grouping

ronya — Wed, 05 Aug 2020 07:52:28 GMT

Thanks Ksharp. If I understand correctly we can use either quantile, bucket OR tree method for binning? Is that correct?

The documentation states that quantile/bucket binning is a pre-bin stage before a Tree based method can be applied:

"The Interactive Grouping node first performs binning on the interval characteristic. You can choose between two binning methods: quantile and bucket. The quantile method generates groups. The groups are formed by ranked quantities with approximately the same frequency in each group. The bucket method generates groups by dividing the data into evenly spaced intervals that are based on the difference between the maximum and minimum values.
After the interval variables have been pre-binned, a decision tree model is fitted for each characteristic. "

So is tree binning a sequential process starting with quantile/bucket pre-binning or we can use quantile, bucket and tree as alternative binning methods?

Re: SAS Softwares

Reeza — Wed, 05 Aug 2020 02:31:36 GMT

https://www.sas.com/en_us/software/visual-text-analytics.html

Or SAS Text Miner.

SAS Softwares

Sha8 — Wed, 05 Aug 2020 01:42:44 GMT

Hi,

I want to know the SAS softwares that could perform coding and performing a sort of thematic

analysis to identify themes. I know some SAS softwares that identify themes on the basis of number of counts and frequencies of a variable in the big data but I need a sort of coding analysis like Nvivo does and I do not need analysis on the basis of number of counts of a variable.

I want to collect information from the different online platforms that includes social networking sites, customer reviews and companies website. Can anyone identify few SAS softwares that pull the data from different social networking sites and companies website and also SAS softwares for identifying themes like Nvivo does and as well as SAS softwares for the analysis purpose? I also want to conduct semi structured interviews and I want to identify patterns through coding therefore I would be needing SAS software to perform coding and identify the themes. That would be good if anyone could mention the possible list of softwares that meet my research needs as mentioned above. I would really appreciate your assistance.

Thank you,

Kind Regards,

Shabana

Re: Binning and Pre-Binning in Interactive Grouping

Ksharp — Tue, 04 Aug 2020 12:59:18 GMT

quantile or bucket method are simple and easy to use.
If you are using Credit ScoreCard ,Tree-based binning can't guarantee the woe is monotonic .

Binning and Pre-Binning in Interactive Grouping

ronya — Tue, 04 Aug 2020 12:03:58 GMT

Hello all, I'm exploring the use of interactive grouping in SaS EMiner as a method to bin the values of interval characteristic and wish to ask about the pre-binning process. Why do we need to use the quantile or bucket method to pre-bin the interval variable values rather than apply Tree-based binning to the interval values directly?

Free Webinar: How Do I Integrate SAS® Viya® and Open Source?

MelodieRush — Mon, 03 Aug 2020 17:32:08 GMT

Hi Data Mining Community,

I’m presenting a live “Ask the Expert” webinar on August 18, 11 AM – Noon ET. SAS delivers an open analytics platform, built on the latest cloud technology and accessible from the interface or in the coding language of your choice, giving you the freedom to experiment and create. Combining the power of SAS with open source technologies, you can unify disparate tools and analytic assets into a streamlined, collaborative environment – fostering productivity, business agility and tangible results. Join me as I discuss the many ways SAS Viya integrates with open source.

You will learn how to:

Access the power of SAS using your existing skills, like SAS, open source or other programming skills.
Use Python or R in the analytical flow of pipelines.
Use Python or R through the SWAT package.

Click here to register for the webinar.

Want more tips? Be sure to subscribe to the Ask the Expert board to receive follow up Q&A, slides and recordings from this and other SAS Ask the Expert webinars.

Can't join the live event? You can view this and other Ask the Expert sessions on-demand here.

Want more tips? Be sure to subscribe to the Ask the Expert board to receive follow up Q&A, slides and recordings from this and other SAS Ask the Expert webinars.

Can't join the live event? You can view this and other Ask the Expert sessions on-demand here.

Re: How to use an existing score code in a new pipeline (build models viya)

acordes — Wed, 29 Jul 2020 16:59:49 GMT

I have tried your solution at https://communities.sas.com/t5/SAS-Data-Mining-and-Machine/How-to-use-the-scoring-code-from-gradient-boosting/td-p/263498

by inserting the %inc line.

%inc '/caslibs/danalytics/svd.sas';

But it doesn't work as expected due to the score file type giving me an error.

So I tried the following but without success as well.

proc astore;
score data=&dm_data 
rstore=MODELS._654S34YEIM3X7NLUBHJ8PQRK2_AST
out=&dm_data_caslib.test; 
run;

filename delta "&dm_file_deltacode";
data _null_;
  file delta;
  merge &dm_data &dm_data_caslib.test(keep= _SVDDDISTANCE_ bastidor origin);
  by origin bastidor;
  put "if NAME='_SVDDDISTANCE_' then do; ROLE='INPUT'; LEVEL='INTERVAL'; end;";
run;
filename delta;

How to use an existing score code in a new pipeline (build models viya)

acordes — Wed, 29 Jul 2020 15:00:01 GMT

I want to derive a scoring code (i.e. from a cluster that I'd learned before) and assign this new variable an input role or filter role.

So I try to insert a sas code node and run the proc astore that works fine in SAS Studio.

But how can I achieve that the new variable shows up in the training data set?

/* SAS code */
proc astore;
score data=&dm_data
rstore=MODELS._654S34YEIM3X7NLUBHJ8PQRK2_AST
out=&dm_data_outmodel; 
run;

Re: How to recover a diagram from SAS miner

Jan3 — Mon, 20 Jul 2020 14:53:22 GMT

Hi, may I know if you found the solution for this missing diagram? I am facing the same problem and it is urgent. Thanks

Setting variable weight on decision tree using SAS Miner

Livia_L — Tue, 14 Jul 2020 20:14:12 GMT

I have a data set with a binary target where only 1.3% of the members are marked as 1, while 98.7% are marked as 0, and I need to run a decision tree on that set. So, I tried two ways of doing that:

1) I created a simple random sample on SAS with 30% of the members marked as 1 and 70% marked as 0, and also created a "weight" variable on the data set, where the weight of the members marked as 1 was 1 and the weight of the others was 22 or something like that. Then, when defining my data source on SAS Miner, I assigned the frequency role to my weight variable. I tried to run the decision tree on that set, but it only returned one node

2) I read the Decisions Node Example from this page https://documentation.sas.com/?docsetId=emref&docsetTarget=p0ak10c0ywr8mrn1an7tum03sp7g.htm&docsetVersion=14.3&locale=em and tried again, following the instructions. I set the "Decisions", and then connected it to the "Decision Tree", and on Train Properties, I defined Import Tree Model as Yes, runned the tree, and got an error

What I'm doing wrong?

Re: How to use validate and test datasets manually in PROC GRADBOOST?

BrianGaines — Thu, 09 Jul 2020 20:28:58 GMT

Hi ,

Are you using SAS Studio to develop your code? If so, then I suggest that you take a look at the tasks (specifically, the Partitioning, Gradient Boosting, and Assess tasks) because they can expedite your code development.

As mentioned, a convenient way to organize your data is to have one data table with an indicator variable that denotes which partition an observation belongs to. One benefit to this approach is that, when you estimate your model and use the PARTITION statement, some performance metrics for the validation and test partitions are automatically calculated so you don't need to calculate them separately as an additional step.

For example, the following code creates a CAS session, loads SASHELP.CARS as an in-memory table, and partitions that table into three sets (the PROC PARTITION code is from the Partitioning task):

/* Connect to CAS */
cas;
libname mylib cas caslib="casuser";

/* Load data into memory */
data mylib.cars; 
   set sashelp.cars; 
run;

/* Partition data set */
proc partition data=mylib.cars partind samppct=30 samppct2=10;
	output out=mylib.cars;
run;

Now the data table MYLIB.CARS has a new _PartInd_ column where 0 corresponds to the training set, 1 for validation, and 2 for test.

You can then use this data table with the PARTITION statement in PROC GRADBOOST, as is done with the following code (generated by the Gradient Boosting task):

proc gradboost data=MYLIB.CARS outmodel=mylib.savedModel;
	partition role=_PartInd_ (validate='1' test='2' train='0');
	target Origin / level=nominal;
	input MSRP EngineSize / level=interval;
	input DriveTrain / level=nominal;
	ods output FitStatistics=work.Gradboost_fit;
	score out=mylib.scored copyvars=(Origin MSRP EngineSize DriveTrain _PartInd_);
run;

You can see in the results that the procedure automatically calculates fit statistics for all three partitions:

You could also use the saved model (mylib.savedModel) and PROC GRADBOOST to score the validation set, like in the following code:

proc gradboost data=MYLIB.CARS(where=(_partind_=1)) inmodel=mylib.savedModel;
	output out=mylib.valscored copyvars=(_all_);
run;

And you can see that the fit statistics match those produced by PROC GRADBOOST for the validation set when you estimated the model (compare with the previous results):

But again, by organizing your data partitions into the same table and by using the PARTITION statement, SAS automatically calculates these fit statistics when you estimate your model. You can also use the scored data table (mylib.scored) with the Assess task for additional model assessment.

Does this help?

-Brian

Re: How to use validate and test datasets manually in PROC GRADBOOST?

StatDave_sas — Thu, 09 Jul 2020 15:51:56 GMT

Concatenate your separate data sets into one data set with an added variable that has a distinct value for the training, validation, and testing sets of observations. Then add a PARTITION statement in your PROC GRADBOOST step. For example, if the added variable is named ObsType with values "trn", "val", and "tst":

partition role=ObsType(train='trt' validate='val' test='tst');

See the documentation for details on this statement.

How to use validate and test datasets manually in PROC GRADBOOST?

mh2t — Thu, 09 Jul 2020 15:43:20 GMT

I did split my dataset into 3 separate sas datasets train, validate and test.

I wanted to build a GBM model on the train set, check on the validate set and predict on the test set. How can I use the validate and test sets in my code explicitly for model checking and prediction?

proc gradboost data=mylib.train outmodel=mylib.savedModel seed=12345;
   input &myVars / level = nominal;
   target Y/ level = nominal;
   ods output FitStatistics=fitstats;
run;

Your help would be greatly appreciated!

Re: Tracking Social Distancing Using Computer Vision

Patrick — Mon, 06 Jul 2020 13:52:16 GMT

Thank you for your response and I have no doubt that SAS is doing the right thing for the right reasons. Having said that: Your post made me realize that the PII data question is of a much more complex nature than I've been aware of so far. My current question is: Is a picture/video of me PII or not? And is it already PII as such or only if "someone" publishes the picture or uses facial recognition or the like on it. And then taking GDPR regulations: Does this now mean that I could request from all persons/companies/agencies operating security cameras in Europe to delete me from any pictures/films they've taken from me anywhere I've been in Europe? And where is the list of these cameras and its owners? Feels like tons of unresolved practical and ethical questions.

Aircraft Turnaround Management Using Computer Vision

mentos05 — Mon, 06 Jul 2020 13:32:29 GMT

In this article I will demonstrate how Computer Vision can support the process of turnaround management in the aviation industry. Therefore I’ve developed a demo application and decided to share my thoughts I had during the development.

But first let’s clarify what turnaround management really is by learning from the experts:

Turnaround management, when applied to the aviation industry, refers to the physical process of preparing an aircraft for its next flight […]. The process of turnaround management is addressed when an aircraft is on the ground — from the time it arrives at a terminal gate, to the time it departs on its next scheduled flight.

-Collins Aerospace Airport Knowledge

In my simple words, turnaround management is everything that has to be done on the ground to prepare the aircraft for the next flight including boarding, baggage loading and fueling.

This process is crucial for every airline and should be optimized in every way possible to reduce downtimes, save costs and make customers satisfied with in time flights.

One of the main problems optimizing the process is the lack of actual knowledge about the current performance in the fleet. In this demo I show my way of using Computer Vision to extract this knowledge from camera streams at airports.

I am going to use different software products for different tasks:

SAS Viya to develop an object detection model
SAS Event Stream Processing to analyze the video in stream (also called SAS ESP)
Python interfaces from SAS (SAS SWAT, SAS DLPy, SAS ESPPy) to communicate with the SAS software
Open Source libraries like OpenCV, Numpy and matplotlib for analytical and visual tasks, e.g. polygon calculation and visualizing results

As you can see, this work is a combination of powerful SAS software and easy-to-use open source packages in Python. Why choose one if you can have both? 🙂

But now let’s get our hands dirty!

1. Define your business processes

Yes, everyone loves training computer vision models and visualize the results. But before starting your precious GPUs you have to decide what business processes you’re interest in and how they are defined.

I am by no means a turnaround expert so don’t expect too much from my definitions. You can define them as complex as you need them. For this demo I kept it simple, e.g. : If I have at least 2 people on the stairways I assume that the boarding has started.

Aircraft Turnaround Task Definitions

Defining these business processes will support you in the next two parts.

2. Define the objects you want to detect and train a model

Now, given the process definitions it will be easy to create a list of objects you want to detect. In my case I decided to train the model for the following objects: aircraft, person, baggage truck, ramp loader, bus, fuel truck, tank hose, ground power, stairway and rolling stairway

Yes these are more objects than I have in my process definitions but I thought I could maybe use some of them later. 🙂

If you’re unlucky like me and don’t have access to training data you can try to find videos on YouTube. For my example I used this video and labeled it using CVAT.

Finally, yes, now is the time for your GPU power!

For object detection I used a standard Tiny YOLO V2 model but other object detection models are possible. The training process is straight-forward but you can have a look at it here.

3. Define Areas of Interest

With your model being able to detect relevant objects you should now go back to your process definitions and extract the relevant areas for your business process. For example you don’t want to count the guys lifting your heavy baggage into the aircraft as passengers boarding the aircraft.

Looking at my definitions, I’ve decided to create the following five Areas of Interest: aircraft area, baggage area, stairway 1, stairway 2, fueling area

Areas of Interest and their corresponding Task Descriptions

For demo purposes I decided to manually define the Areas of Interest. However, you could of course define them dynamically given the objects you detect. This way the application would be more general, e.g. to fit other aircraft types or airports.

4. Create your Turnaround Management application

Now comes the fun part and we are putting all the pieces together to develop our Computer Vision aided Turnaround Management application.

As said in the beginning I’ve used SAS Event Stream Processing to create the image analysis process and afterwards connected this stream to SAS Viya to serve a simple SAS Visual Analytics dashboard showing a turnaround timeline.

From Streaming Data to Live Dashboarding

Turnaround Image Analysis in SAS Event Stream Processing

The SAS ESP process is very standard at the beginning. We are loading and deploying or computer vision model and provide it to the scoring window. Additionally we are receiving the camera image and resize it to fit the needs of the YOLO v2 model.

The YOLO v2 model is applied to the images and we receive the object coordinates. These object coordinates are then fed into a calculation window which will apply our process definitions and produce the visualizations.

Because I am a big fan of programming in Python, I decided to code the business rules in Python too. This is not a problem because the calculation windows in ESP allow us to run Python code via SAS Micro Analytics Services. There are even “PythonHelper Windows” in the SAS ESPPy interface to SAS ESP where you only have to provide the location of your code making the whole process very simple.

The calculation window produces one line of data per frame, including the scored image and several timestamp variables. I’ve used OpenCV to display the scored frames and also coded a small and ugly “dashboard”. Having this dashboard next to our image allows us to check whether all processes have been identified correctly.

Analyzing the video stream of a Turnaround - Visualized output of the SAS Event Stream Processing Pipeline

Turnaround Dashboard in SAS Visual Analytics

Even though looking at the results gives us already a feeling of being done, this is usually not the case. Applying computer vision models to your images is useless if you don’t make use of the information they extracted from images.

Therefore I decided to use the SAS Cloud Analytic Services Adapter to push the results into my SAS Viya environment. This allows me to use the data directly for further analysis, e.g. creating nice reportings or even do more sophisticated stuff like predictive modelling or forecasting.

All I had to do is to add a transpose window at the end of my SAS ESP process that creates the nice and small 3-column table that you can see in the diagram.

The rest is simple dashboarding as you would expect it from professional reportig and analysis software.

I am not a great report designer but you can have a look at my example report here:

Streaming the analyzed data to SAS Visual Analytics for Live Dashboarding

On the left side I used a so called “Data -Driven Content Object” that links to a simple websocket connector written in JavaScript to grab the scored frames from my ESP process. (Yes you can connect to ESP via a lot of different ways!) The good thing about a Data-Driven Content Object is that you can not only receive data but also send data to it directly from VA. This would enable us to interactively change the connection to a different ESP process that observes another aircraft.

The left side uses the transposed data coming from the ESP CAS adapter to create a simple timeline chart. The report refreshes its data every second to accurately show the current process status.

With the data being available in SAS Viya for all kinds of analysis, I’d like to finish this article and would love to hear about your thoughts.

Michael Gorkow | Data Scientist @ SAS Germany & CV-Enthusiast

LinkedIn | GitHub | Medium.com

Crime Investigation using Image Data in the times of Big Data

mentos05 — Mon, 06 Jul 2020 13:09:47 GMT

Social media platforms such as Facebook, WhatsApp and WeChat are wonderful ways to communicate with people all over the world.
Unfortunately these platforms are also used by criminals which is the reason why crime fighters want to investigate the contents on these platforms.

While these platforms have stored a lot of metadata which is more or less available (especially for government organizations), the most interesting content is of course the one that is created by the users. This content usually consists of unstructured data such as text, speech, videos and images. In this article I am focusing on the latter, images, but the approach itself can be used for all kinds of unstructured data.

Our task: Enabling efficient ways of investigating image data

Now, imagine being part of an investigation group of the police that is responsible to fight weapon crimes, right-wing or terroristic organizations.
You have scraped data from various channels, e.g. Twitter, Facebook, WeChat & Co. and now you’re sitting on a huge pile of images not knowing where to start looking.
Manually investigating these data sources is impossible, due to their size and variety.

Our goal is to provide an efficient way of filtering this data, allowing investigation officers to focus on the relevant images and their sources. This not only saves manual effort but also enables officials to better protect the law.

The idea: Image classification to create additional metadata

The idea is simple. We will train an image classification model that is able to classify our images into various categories. So to speak, we generate additional metadata for our images.

This classification is then combined with other data we already have such as source, date, location, etc.

Finally we want to create an intuitive dashboard that enables investigation officers to quickly identify potentially interesting content and skip through all the nice and funny images of cats. 🙂

As soon as we have the trained model available, the workflow is the following:

Metadata coming from new Images is directly fed into the analysis environment such as SAS Viya.
Images are scored using our computer vision model
Classifications and Metadata are combined

The combined data from step 3 is then fed into a visual report allowing crime investigators to perform in-depth analysis.

Getting our hands dirty: Training the image classification model

I am using the deep learning capabilities of SAS Viya to develop the image classification model. Of course you can develop the model using the SAS programming language but I am more comfortable using Python and Jupyter Notebooks. Luckily there are powerful Python-APIs that allow you to interact with your SAS environment. I am using the following APIs

SAS SWAT (communicate with SAS Viya)
SAS DLPy (special deep learning API for SAS)

In this Jupyter Notebook you can see how I use transfer-learning to train a ResNet-50 image classification model that afterwards can detect different weapons. The model was pretrained on the ImageNet dataset which contains images from 1000 classes and can be downloaded here.

There is also a second example where I apply the same process to images from hate symbols such as ISIS flags or swastikas.

The data used for training comes from two different sources:

Kaggle Weapon Dataset
Hate symbols that I scraped from Google Image search by hand

Here is an example how my training data looks like for the weapon detection:

The Jupyter Notebook comes with a lot of comments that explain in-depth what I am doing. Therefore I am not going into the details here to keep the article as short as possible.

After training the model, I evaluated its performance using a confusion matrix. As you can see the predictions are quite accurate with an accuracy of 99%. This seems a little bit too high for me and might be due to the training data being really nice to us. A lot of images have a nice white background which you cannot expect from real social media images. 🙂

Finished? No! A nice model is useless for our investigation officers.

A lot of articles about Computer Vision end when the model was trained and successfully scored some new data. However if you want your efforts to be useful you also have to think about how to put your models into the bigger picture. While SAS has advanced technology for this, e.g. the SAS Model Manager, I am not going into the details of production-ready applications here.

Instead, I’d like to demonstrate how a simple dashboard that uses our model’s predictions could look like. The dashboard was built with standard functionality of SAS Visual Analytics.

Example 1: Investigate Images of Weapons coming from Social Media

Example 2: Detect Hate Symbols on Images coming from Social Media

The only interesting part is the “Data-Driven-Content Object” that is responsible for displaying the selected images which you can see on the right side of the reports. This object dynamically receives the data selection from the other report elements. Behind the scenes it is using a very simple JavaScript code to retrieve and display the selected images.

As you can see in the videos our model acts as a supplier of additional metadata. It allows us to use the filter in the top-left corner to select the image categories we are interested in.

This use case does not only apply to crime investigation but is much more general.

Even though this article focused on identifying crime there are of course many other interesting use cases. Another idea could be that you want to identify your brand’s logo or products in social media posts. This way you could for example promote authentic user postings that highlight advantages of your products or you can try to give additional support if it is a customer complaining.

Do you have other use case ideas for this approach? I’d like to hear about them in the comments!

Michael Gorkow | Data Scientist @ SAS Germany & CV-Enthusiast

LinkedIn | GitHub | Medium.com

Re: Tracking Social Distancing Using Computer Vision

mentos05 — Mon, 06 Jul 2020 12:45:19 GMT

Thank you Patrick for your comment and I agree with you that one can imagine scary scenarios using Computer Vision technology.

Therefore, we as a society, have to make sure that we use such technology for good things - not bad things.

Speaking for this particular demo, the model used is a Tiny YOLO V2 network that is not able to recognize entities. It can only detect people in general and there is no plan from my side to enrich it with face recognition.

Thinking further, I would say that such systems can also help to ensure data privacy. After analyzing the video and calculating key figures (number of rule violations, people-density-index, etc.), we could immediately delete the video as we are usually not interested in the video itself but its condensed information that was extracted by computer vision.

Best Regards

Michael

Re: Tracking Social Distancing Using Computer Vision

Patrick — Mon, 06 Jul 2020 13:58:39 GMT

That's very cool and very scary at the same time. Add facial recognition and weak privacy laws to it and 1984 becomes reality.

Or even just racial or social crowd profiling... I guess the privacy discussion about pictures, security cameras and its uses and data storage becomes more and more urgent. Who "owns" a picture of me? And what are you allowed to do with it? Hoping for an amendment to GDPR and the like!

Tracking Social Distancing Using Computer Vision

mentos05 — Mon, 06 Jul 2020 13:11:02 GMT

Due to the recent pandemic, several governments have decided to implement restrictions for social distance. While most people follow these guidelines, there are still people who ignore them for various reasons.

The overall goal of this project is to identify objects (i.e., people) who are following social distancing guidelines and those who are not. This is accomplished in the following way:

Transform images and object coordinates into a two-dimensional map if a homography matrix was provided
Calculate real world distances between detected objects if a homography matrix was provided. Otherwise use distances from the image directly.
Detect crowds that exceed a specified parameter
Visualize the results on the camera image

Object Detection

The first step in the process is to detect people in images. A Tiny Yolo V2 model was used for this task, but any detection model would work. The decision to use Tiny Yolo V2 was mainly driven because of its ease to use in SAS.

Transforming the Image and Calculating Distance

The second step is to calculate the distances between all objects. While this may sound simple, it is more complicated if you think about it. Normally a camera does not provide a top view. Instead it is at some angle which leads to certain perspective. This perspective can be very important when you calculate real world distances in images. The following three pictures illustrate the transformation process.

The second image shows the results after transforming the image using a Homography Matrix. A rectangle was able to be formed, but the distances are incorrect due to the camera's perspective. The third image show the results after a transformation matrix was applied. A distance of 356 pixels on the image shrinks to 149 pixels while a distance of 62 pixels in the other direction are 83 pixels in reality.

Identifying Crowds

Now that we have detected people and transformed the distances between them we want to determine whether we have crowds in our image. In this demonstration, KD-Trees from Scipy was used to look up nearest neighbors given a detected person and a maximum radius. The following functions were used:

cKDTree to efficiently calculate nearest neighbors.
query_ball_tree to query the KDTree with a person and a given maxium radius.

Streaming Process

The streaming process uses SAS Event Stream Processing (ESP) and can connect to any video data source. ESP lets you define the process either graphically or programmatically via a Python Interface.

The following diagram illustrates the streaming process:

The top two boxes load the trained Tiny YOLO V2 model and provide the model to the scoring window. The scoring window receives images that are resized to appropriate dimensions (416x416 pixels in this case). The scoring window provides the detected persons and their corresponding x, y, width, and height values.

The last box utilizes the Python inside SAS Event Stream Processing to transform the coordinates given the homography matrix. Additionally, it uses Scipy to perform crowd detection.

Ressources & Results

For a more detailed look (incl. code) at Tracking Social Distancing Using Computer Vision, refer to this GitHub page.

And now let's view the results!

The left side shows the original camera image that was annotated by our streaming process. On the right you can see the top view produced by our homography matrix. Green colors show long distances, orange and red indicate small distances. Last but not least blue squares are drawn around crowds of three or more people.

Michael Gorkow | Data Scientist @ SAS Germany & CV-Enthusiast

LinkedIn | GitHub | Medium.com

Re: VDMML sas code node not affecting output data

Ullsokk — Wed, 01 Jul 2020 12:35:37 GMT

Figured out I could use the macro %dmcas_metaChange to change the role to rejected, as it seems variables cannot be dropped in the code node

VDMML sas code node not affecting output data

Ullsokk — Wed, 01 Jul 2020 08:40:17 GMT

I have a simple code snippet I want to run in the SAS code node in VDMML, that removes varaibles with given suffixes. Here is my code:

proc contents noprint data=&dm_data out = contents (keep=name) ;
run;


data contents;
	set contents ;
	if index(lowcase(name),'_dt') then drop=1 ;
	if index(lowcase(name),'_rk') then drop=1 ;
	if index(lowcase(name),'_dttm') then drop=1 ;
run ; 
	
proc sql;
	select name into: dropvars separated by ' ' from contents where drop = 1;
	quit;

data &dm_data;
	set &dm_data (drop=&dropvars);
run;

The proc contents and SQL seems to work, and I get no errors in the log. The log even implies that the final dataset has 1602 variables, down 7 from the original 1609, just as expected. But the column is still there in the results view. And in the next node, the column is still visible, and populated with data.

Am I using the wrong macro variable name for the data? Do I need to do a CAS action? or do I have to do some metadata update?

Ddos Attack

abhishekpandey — Fri, 26 Jun 2020 07:54:34 GMT

Which machine learning is best for detection of ddos (Distributed Denial of Service Attacks) attack ?

If there is options to select one of the following.

1). Performance evaluation of Botnet DDoS attack detection using machine learning.

2). Smart Detection: An Online Approach for DoS/DDoS Attack Detection Using Machine Learning

3). DDoS Attacks Detection Using Machine Learning Algorithms.

4). Machine Learning Based DDoS Attack Detection From Source Side in Cloud.

5). DoS Attacks Intrusion Detection Algorithm Based on Support Vector Machine.

Event-based sampling in Viya

ravnen55 — Wed, 24 Jun 2020 13:31:43 GMT

I have a bit of a weird problem:

I constructed a project in model studio in Viya, and used event-based sampling. The different models all had Roc/accuracy curves that looked something like this:

Great, that is what I expected.

Now I had some issues with modelmanager, and I had to reconstruct the project. Using the same dataset, the same models and again using event-based sampling, the curves now looked like this:

The probabilites were now around 95 % for an event=1, whick was in accordance with the distribution in the original dataset but NOT what i would expect after having done evet-based-sampling.

Obviously I must have done something different in the second project...But any ideas of WHAT box I might have ticked or what I might have otherwise done, that can cause this difference, would be much appreciated

thx

Re: K-fold Cross-validation of Neural Networks

dcfroehlich_aol_com — Wed, 24 Jun 2020 11:44:41 GMT

Does it matter? Assume that I am both.

Re: K-fold Cross-validation of Neural Networks

WendyCzika — Tue, 23 Jun 2020 15:32:20 GMT

Hi there - are you on SAS 9 (Enterprise Miner) or SAS Viya?

K-fold Cross-validation of Neural Networks

dcfroehlich_aol_com — Sat, 20 Jun 2020 05:56:42 GMT

My question concerns the use of K-fold cross validation for artificial neural networks (NN). Specifically, I want to know where the final NN model parameters come from? Were they obtained by a fit to the entire data set? Or, were they from a fit to one of the K-1 fold data sets used for training. The SAS documentation is not clear on this issue. Does anyone have an answer?

Re: SASMiner - Cannot connect to "SASMiner - Logical Workspace Server"

Tom007 — Fri, 19 Jun 2020 12:55:26 GMT

Restart of entire SAS application services helped issue to fix.
Also, Mid-tier restart would be helpful.
Thank you Anand_V much appreciated for your assistance.

Re: How do I subset a large dataset using a list of specific ID numbers?

novinosrin — Thu, 18 Jun 2020 21:19:02 GMT

 Data want;
SET have;
where patientnum in (3 6 5 4 8 7 9 0) ;
run;

/*or*/

where patientnum in (3, 6, 5, 4, 8, 7, 9, 0) ;

How do I subset a large dataset using a list of specific ID numbers?

sasgyro — Thu, 18 Jun 2020 21:08:39 GMT

I have a series of patients who have certain characteristics( variables)

I need to pick out the patients that meet the criteria by using their patient ID (300 unique identifiers) and separate them into their own dataset so they are alike only by that one characteristic.

i=I want it to be like this:

Data want;
SET have;
where patientnum= 3 6 5 4 8 7 9 0 ;
run;

but this doesn't work. and using OR and AND statements cancels the previous ID number

Text Mining

Ghabek — Thu, 18 Jun 2020 16:55:39 GMT

Hello All

Im using an old dataset from the text miner days in 9.4 to show subrogation using insurance adjustor notes to improve the model accuracy. There are some categorical variables but I cant find the mapping of the levels for the fields.....I guess I can make it up but would rather get the actual data definition if they actually existed!!

Re: What should be the Optimum Number of Cluster

MelodieRush — Thu, 18 Jun 2020 13:37:27 GMT

Aligned Box Criterion is available in the HP Cluster node in SAS Enterprise Miner. It will determine the optimum number of clusters. Here's a video that talks about using this option, along with using CCC and gap methods https://www.youtube.com/watch?v=NZpNTkfT47c

Re: What should be the Optimum Number of Cluster

arpitsharma27 — Thu, 18 Jun 2020 13:24:19 GMT

Thank You &

I know _ccc_ is strictly negative.

I am already using PCA too.

The idea is to get outliers from 2 different algorithms and then join to get the output.

PCA-- was able to handle this.

But

KNN-- is looking for better selection of variables. Just dumping variables for KNN to figure out the cluster does not seem to be the correct thing to do.

Thank You to the Legends.

Re: What should be the Optimum Number of Cluster

Ksharp — Thu, 18 Jun 2020 12:19:15 GMT

It is a world unsolved problem.

If I were you, I would try Primary Component Analysis.

Anyway, maybe have some ideas .

Re: What should be the Optimum Number of Cluster

PaigeMiller — Wed, 17 Jun 2020 18:28:14 GMT

You are getting only negative numbers for CCC. This implies (to me) that there is no clustering. Also see https://www.researchgate.net/post/Could_someone_help_me_decide_the_ideal_noof_clusters_from_the_pseudo_t_squared_graph_in_SAS

which says

CCC is the cubic clustering criterion; the idea behind it is to compare the R squared you get with a specific number of clusters versus the R squared you would get by clustering a uniformly distributed set of points. That is, you interpret it similarly as you would R squared. You are getting STRICTLY negative values (and, in fact, they are decreasing with additional number of clusters before increasing again; I would interpret that increase as overfitting). This means that the model you are fitting to the data with X number of clusters fits worse than uniformly distributed points. This is evidence of a lack of clustering (or problems with the data).

What should be the Optimum Number of Cluster

arpitsharma27 — Wed, 17 Jun 2020 17:26:16 GMT

Team,

I have this:

%macro clustering_method(method=);
proc cluster data=cars method=&method. ccc outtree=tree_&method. noprint;
where type="Sports";
by type;
	var horsepower mpg_highway weight wheelbase;
run;
proc sort data=tree_&method. out=&method.(keep= type _ncl_ _ccc_ );
by type _ncl_ _ccc_ ;
where not missing(_ccc_);
run;

%mend;
%clustering_method(method=Average);
%clustering_method(method=median);
%clustering_method(method=centroid);
%clustering_method(method=mcquitty);
%clustering_method(method=ward);

data Have;
set Average 
	Median 
	Centroid 
	McQuitty 
	Ward 
		indsname=source;
input_ds=scan(source,2,'.');;
run;

Referring to the Have dataset. What should be my optimum number of Clusters ? and Why?

Please advise.

Thanks

Re: Gini performance metric for decision tree

Scott86 — Wed, 17 Jun 2020 00:08:28 GMT

It is in the model comparison node under assess for anyone else who has the same questions

Gini performance metric for decision tree

Scott86 — Tue, 16 Jun 2020 23:52:38 GMT

Hi everyone,

I am building two classification decision tree in SAS enterprise miner. I'd like to compare the performance of the decision trees with a Gini metric. How do I get performance stats when using SAS enterprise miner? If I can't get Gini can I get ROC or AUC and then I can calculate Gini.

Thank you for any help

Miner interactive binning export

miner_girl — Fri, 12 Jun 2020 12:25:21 GMT

Dear SAS community,

I need help regarding export of interactive binning node, part of SAS EM.

I've split variables in groups and know I would like to observe their monotonic trend.

This view is very helpful and easy to explain to others. Now, I would like to export this view and then use it for presentation or for documentation. Do you have any idea how it can be managed?

I've tried reporter node, but it can draw only 25 variables and bucket's boundaries are below in table, therefore it makes it harder to interpret.

Thank you in advance for all suggestions.

Model comparison in SAS VS How does this work

PaalNavestad — Thu, 11 Jun 2020 13:56:40 GMT

Hi, I'm using the model comparison in SAS/VS. It does some strange things. I have an RF model with reported Validation ASE of 197,611. I have a Gradient Boosting (DSG) model with a reported Validation ASE of 199,686. After running this through model comparison I get a validation ASE of 196,927 for the RF model and 202,275 for the DSG model. These numbers vary with the number of models being in the Model Comparison.Random forest variable importanceGradientBoosting variable importanceModel comparison HeatmapModelcomparison table

Of a larger concern is that the Variable importance is completely changed. The variable that in the DSG model is the second most important has suddenly become the 7 most important after Model comparison. Also the variable importance HeatMap is inconsistent with the variable importance table.

Re: variable clustering dataset

chmedi — Tue, 09 Jun 2020 17:20:11 GMT

Since you're a SAS Partner, why don't you work through your Partner channels? In the meantime, I have some simulated data that produces three clusters.

Log-into SAS Studio on the same system as Model Studio, and submit the following code to generate the data in Public:

cas;
caslib _ALL_ assign;

%macro makeRegressorData(nBy=1,nByFixedSize=1,nObs=100,nCont=4,
nClass=3,nLev1=3,nLev2=5,nLev3=7);
data testdata;
drop i j;
%if &nCont>0 %then %do; array x{&nCont} x1-x&nCont; %end;
%if &nClass>0 %then %do; array c{&nClass} c1-c&nClass;%end;

do by=1 to &nBy;
if by > &nByFixedSize then
nObsInBy = floor(2*ranuni(1111155)*&nObs);
else nObsInBy = &nObs;
if nObsInBy < 10 then nObsInBy = 10;

do i = 1 to nObsInBy;
%if &nCont>0 %then %do;
do j= 1 to &nCont;
x{j} = ranuni(1);
end;
%end;

%if &nClass > 0 %then %do;
do j=1 to &nClass;
if mod(j,3) = 0 then
c{j} = ranbin(1,&nLev3,.6);
else if mod(j,3) = 1 then
c{j} = ranbin(1,&nLev1,.5);
else if mod(j,3) = 2 then
c{j} = ranbin(1,&nLev2,.4);
end;
%end;

weight = 1 + ranuni(1);
freq = 1 + mod(i,3);

*** if ( i = 11 ) then x{2} = .;
*** if ( i = 12 ) then c{1} = .;

output;
end;
end;
run;
%mend;

%macro AddDepVar(modelRHS =,errorStd = 1);
data testdata;
set testdata;
y = &modelRHS + &errorStd * rannor(1);
run;
%mend;

%makeRegressorData(nBy=1,nByFixedSize=1,nObs=435,nCont=14,
nClass=11,nLev1=1,nLev2=3,nLev3=3)
/* %AddDepVar(modelRHS=2*c1 - 0.5*x1,errorStd = 1); */

data testdata2(drop=x105 x106);
set testdata;
x105 = x5;
x106 = x6;
x109=x9;
x5 = x2 + 0.49147127158235 * x3 -0.5868629022532 * x4 - 1.7447414951792 * x105;
x6 = x1 - 0.13846905761895 * x3 + 0.6800302223946 * x4 - 0.45238025995657 * x105;
x4 = x106 + 0.00146547818375 * x1 + 0.03017986135363 * x2 - 0.00820230703132 * x3 +
0.03457381414136 * x4 + 0.03738296997610 * x105;

x9 = x7 + 0.7138912 * x8 - 0.45238025995657 * x109;
x10 = x8 / 30 + 0.931261126 * x7+ 0.03738296997610 * x109;

run;

data public.test009 (promote=yes keep=x1-x10 c1);
set testdata2;
run;

Then, switch to Model Studio (Build Models), create a project with test009 (you will need to refresh the available table list). Target=c1. For the Variable Clustering node, the one
change to make to the properties is to set Maximum number of iterations (under Advanced Options) to 30.

Re: variable clustering dataset

Ghabek — Tue, 09 Jun 2020 16:06:12 GMT

im a sas partner

how do that

who do i talk to

i used to work in education but left sas

Re: variable clustering dataset

chmedi — Tue, 09 Jun 2020 16:03:01 GMT

Yours is a different data set than that provided in the SAS course. If you could, please contact SAS to request the data.

Re: SAS EM Leveraging - HP Text Miner + HP Forest

Ritwick — Tue, 09 Jun 2020 14:59:27 GMT

Do I need to make two separate models, one for HP Textminer and the other for HP Forest, then combine them?
Currently I am getting the output variables as attached below.
The "COLs" are the SVD clusters... other variables are not selected for building the trees?

Re: variable clustering dataset

Ghabek — Mon, 08 Jun 2020 23:51:24 GMT

Hello

I ran it with your settings and still get one cluster for the variables....Im using Average Gift as the interval target to run the pipeline with the variable clustering node and your settings.

I get one cluster

I got the datatset from the web of 7600 records.......

any ideas why I cant get three clusters?

thanks!!

Re: variable clustering dataset

Ghabek — Mon, 08 Jun 2020 21:59:11 GMT

Great
Can you send me the dataset pls?

Re: variable clustering dataset

chmedi — Mon, 08 Jun 2020 21:54:39 GMT

Hi Ghabek. You can get three clusters with the data set pva_raw_data. This data is used in the EM Applied Analytics course. I had to play with the Variable Clustering node properties to get the three clusters. My changes from default: Use default maximum number of variables per cluster: De-select checkbox, Number of variables per cluster upper threshold = 8, Clustering Rho value = 0.7.

SAS EM Leveraging - HP Text Miner + HP Forest

Ritwick — Mon, 08 Jun 2020 21:01:00 GMT

Hi Folks,

I am trying to build a churn model with the attached model flow. The flow worked well when using the test data. "HP Forest &SVD" node got selected in the Model Comparison. But when I am scored the selected model with the new dataset, the score node is only considering the HP Text Miner output. Let me know how I can resolve this issue.

Re: Case when multiple condition in expression

Tom — Sun, 07 Jun 2020 18:21:53 GMT

Why would you but an ELSE between them? ELSE is for catching all of the observations that do not meet any of the earlier WHEN conditions. If you have multiple WHEN conditions just list then one after the other.

case when ENTITY_ID_SF in ('1000','1111') and SEGMENT='NL' then sum(GR_INPUT)/(sum(PREM_RES_GR)/3)
     when ENTITY_ID_SF in ('1000','1111') and SEGMENT='PL' then sum(GR_INPUT)/(sum(PREM_RES_GR)/3) 
end

Right now both of your two different conditions will return the same result when they are true. So you could collapse them into one WHEN condition. Like this:

case when (ENTITY_ID_SF in ('1000','1111') and SEGMENT='NL')
       or (ENTITY_ID_SF in ('1000','1111') and SEGMENT='PL') then sum(GR_INPUT)/(sum(PREM_RES_GR)/3)
end

or like this:

case when (ENTITY_ID_SF in ('1000','1111') and SEGMENT in ('NL','PL')) then sum(GR_INPUT)/(sum(PREM_RES_GR)/3)
end

But what are you actually trying to do? Did you expect to get different results based on the value of SEGMENT? If so then show some simple input data and what output you want.

Re: Case when multiple condition in expression

ballardw — Sun, 07 Jun 2020 17:33:04 GMT

Please. Anytime you get an ERROR, copy the entire procedure with the messages from the LOG and show the code plus the errors. Please.

Include the entire procedure because the cause could be something like a missed or extra quote or parentheses in a previous statement than the place that SAS indicates the error.

Re: Case when multiple condition in expression

Jagadishkatam — Sun, 07 Jun 2020 14:52:36 GMT

try the below code

CASE
    WHEN condition1 THEN result1
    WHEN condition2 THEN result2
    WHEN conditionN THEN resultN
    ELSE result
END as variable_name 



case 
when ENTITY_ID_SF in ('1000','1111') and SEGMENT='NL' then (sum(GR_INPUT)/(sum(PREM_RES_GR)/3)) when ENTITY_ID_SF in ('1000','1111') and SEGMENT='PL' then (sum(GR_INPUT)/(sum(PREM_RES_GR)/3)) else . 
end as variable_name

Case when multiple condition in expression

David_Billa — Sun, 07 Jun 2020 12:50:00 GMT

I need help writing the case when with multiple conditions to use in 'expression' in DI Job. I'm getting an syntax error if I use below. How to keep adding the conditions in case when? Shouldn't we use 'else' to seperate the condition?

case when ENTITY_ID_SF in ('1000','1111')  and SEGMENT='NL' then (sum(GR_INPUT)/(sum(PREM_RES_GR)/3)) 
end
else
case when ENTITY_ID_SF in ('1000','1111') and SEGMENT='PL' then (sum(GR_INPUT)/(sum(PREM_RES_GR)/3)) 
end

Re: ERROR: Local CASLIB quota exceeded.

Patrick — Sat, 06 Jun 2020 22:07:59 GMT

There seems to be some quota setting so that a process can't take down the server. Documentation here.

Global caslib Policy

Because global caslibs apply to all users, global caslibs have a unique policy.

POLICY DEFINITION

globalCaslibs
   _ALL_         quota
   [global-caslib  quota]
   [...]

global-caslib

specifies a global caslib.

quota

specifies the maximum amount of disk cache space, in bytes, that global-caslib can use.

_ALL_

specifies all global caslibs.

When _ALL_ is specified, quota specifies the total amount of disk cache space that can be used for all global tables, regardless of the caslibs to which the tables belong.

Example

  globalCaslibs 
     _ALL_      400000000 
     HPS        200000000 
     MyGlobal   100000000 

  [CAS-server-priority-n
     cpu               - share
     globalCasuser     - quota
     globalCasuserHdfs - quota
     sessionTables     - quota]
  [...]

Re: Quick way to change the data type of registered table in SAS DI 4.9

Patrick — Sat, 06 Jun 2020 21:49:08 GMT

OK - if you really can't re-create the table then something as painful as below should do the job. Once you've modified the physical table you then can execute an update metadata and re-align the column throughout your jobs (eventually use the standardization wizard if there are too many impacted mappings).

data have;
  var='123';
  output;
  stop;
run;

proc datasets lib=work nolist;
  modify have;
    rename var=_var;
    run;
quit;

proc sql;
  alter table have
    add var float;
  update have
    set var=input(_var,best32.);
  alter table have
    drop _var;
quit;

proc print data=have;
run;
proc contents data=have;
run;

Re: Quick way to change the data type of registered table in SAS DI 4.9

David_Billa — Sat, 06 Jun 2020 21:20:11 GMT

It's a SAS table.

Re: Quick way to change the data type of registered table in SAS DI 4.9

Patrick — Sat, 06 Jun 2020 21:15:44 GMT

'Easiest Way' is to delete the physical table, change your metadata definition, run the DI job that creates the table.

You are right, you can't change the data type with an alter table. Are you dealing with a SAS table or a database table?

Re: Quick way to change the data type of registered table in SAS DI 4.9

David_Billa — Sat, 06 Jun 2020 21:00:00 GMT

From your reply 'Easiest Way' is not possible. Document says

'You cannot use the MODIFY clause to

change a character column to numeric or vice versa. To change a column's data type, drop the column and then add it (and its data) again, or use the DATA step.'

Is there a way that I can change the datatype, without dropping the variable?

Re: Quick way to change the data type of registered table in SAS DI 4.9

Patrick — Sat, 06 Jun 2020 20:52:14 GMT

"So when I tried to change the datatype by going to respective table via inventory and changed the data type..."

Here you changed the table metadata but not the underlying physical table.

"...and did right click and choose 'Update Metadata' "

This updates the table metadata using the underlying physical table - which you haven't changed so now metadata for this column will be character again.

Easiest way: Change table metadata, delete the physical table, run the job which creates the physical table.

If that's not possible: Use Proc SQL/Alter Table; Modify... and change the physical table. Then execute an update metadata

Quick way to change the data type of registered table in SAS DI 4.9

David_Billa — Sat, 06 Jun 2020 20:30:33 GMT

I want to change the data type of one of the variable from character to numeric in one of the table in SAS DI 4.9. While I creating the new table, I had mistakenly created one variable with character data type instead of numeric.

So when I tried to change the datatype by going to respective table via inventory and changed the data type and did right click and choose 'Update Metadata' but still I see the datatype has not changed.

I ensured that I've closed the table before doing this operation. Am I doing something wrong here or is there any other way to change data type in a step or two?

Re: SASMiner - Cannot connect to "SASMiner - Logical Workspace Server"

Anand_V — Sat, 06 Jun 2020 08:08:51 GMT

That's interesting. Could you replicate this issue and capture the object spawner logs? If you don't see anything unusual, try it again after enabling the debug logs.

Re: SASMiner - Cannot connect to "SASMiner - Logical Workspace Server"

Tom007 — Sat, 06 Jun 2020 07:07:49 GMT

Thank you for response Anand_V
I'm able to connect via SAS EG and expand SASApp, able to execute sas program without any issues.

Re: SASMiner - Cannot connect to "SASMiner - Logical Workspace Server"

Anand_V — Sat, 06 Jun 2020 07:04:40 GMT

Are you able to connect to workspace using any other client like EG or SAS Studio?

Re: SASMiner - Cannot connect to "SASMiner - Logical Workspace Server"

Tom007 — Sat, 06 Jun 2020 06:07:39 GMT

Hi Jens89

I hope you are doing good.
How did the issue got fixed, what was the step followed.
I did restart Object Spawner alone, but didn't help.
Stopped and started service on compute server, that also did not work!

Please let me how did you fixed the issue.

ERROR: Local CASLIB quota exceeded.

klroesner — Fri, 05 Jun 2020 14:47:49 GMT

Hello World,

We are trying to run a random forest model in SAS Viya 3.4 VDMML and keep getting this error. Does anyone know what this means? And what is meant by 'quota'? We have not set any OS quota. And maybe one more question where the astore file _6WR5KOEIE2Y4J12PNBZEZAR3Y_AST is written to?

Thanks,

Klaus

NOTE: Writing HTML5 Body file: ods.htm
NOTE: Fileref _FR has been deassigned.
NOTE: The file _TFREF is:
      
      Dateiname=/opt/sas/viya/config/var/tmp/compsrv/default/3246bba6-2709-4bed-b816-b70ff22102a8/SAS_workDA4200000E54_rviap0v01.dst
      .tk-inline.net/74b72ccd-f517-4a62-a07e-53a6196492be/traincode.sas,
      Besitzername=p226526,Gruppenname=tk,
      Zugriffsberechtigung=-rw-r--r--,
      Zuletzt geändert=05. Juni 2020 15.13 Uhr,
      Dateigröße (Byte)=0
NOTE: 19 records were written to the file _TFREF.
      The minimum record length was 3.
      The maximum record length was 93.
NOTE:  Verwendet wurde: DATA statement - (Gesamtverarbeitungszeit):
      real time           0.00 seconds
      cpu time            0.01 seconds
      
NOTE: Using SEED=12345 for FOREST model building.
NOTE: Wrote 10139446832 bytes to the savestate file _6WR5KOEIE2Y4J12PNBZEZAR3Y_AST.
ERROR: Local CASLIB quota exceeded.
ERROR: The action stopped due to errors.

Using interactive grouping labels in custom SAS EM node

tom_evans_79 — Fri, 05 Jun 2020 04:13:59 GMT

I have an Enterprise Miner diagram which does the following: Inputs a dataset -> Partitions into training and validation -> Groups variables using Interactive Grouping -> Creates a credit Scorecard -> Scores up the dataset with the Score node -> Creates a custom report in a SAS Code node.

In the last node (SAS Code), I will calculate various statistics, but essentially I need to output a frequency table for grouped variables, using the group labels created by the Interactive Grouping node. The question is how I access those labels.

To illustrate, here's a very simple piece of code that I could use in the SAS Code node:

proc freq data=&EM_IMPORT_DATA;
	table GRP_age / norow nocol nopercent nocum missing;
run;

The variable 'GRP_age' is created by the Score node using the grouping definitions from Interactive Grouping, and is derived from the input variable 'age'.

This creates the following result:

GRP_age Frequency
--------------------
1 755
2 1190
3 1302
4 1580
5 1100
6 643

...but I would like to see this:

Group Name Frequency
----------------------------

age< 32, _MISSING_ 755
32<= age< 40 1190
40<= age< 47 1302
47<= age< 56 1580
56<= age< 65 1100
65<= age 643

Guessing I need to merge in a table from the IG node. Any ideas what this other table is called (in a way that will work if I connect my SAS Code node to a new Interactive Grouping node)?

Thanks!

Tom.

Re: SASMiner - Cannot connect to "SASMiner - Logical Workspace Server"

Tom007 — Thu, 04 Jun 2020 23:33:54 GMT

What was the solution for this issue?
Object spawner service stop&start didn't help.

variable clustering dataset

Ghabek — Thu, 04 Jun 2020 18:19:03 GMT

Hello

I am still searching for a good dataset showcasing variable clustering in VDMML....maybe showing three or four clusters of variables to reduce multicollinearity on numeric variables.

any assistance would be appreciated!

Re: variable clustering dataset

Ksharp — Wed, 03 Jun 2020 11:37:53 GMT

The classic dataset is SASHELP.IRIS

variable clustering dataset

Ghabek — Tue, 02 Jun 2020 18:20:32 GMT

im looking for a good dataset to do variable clustering on.

Re: Free Webinar: How Do I Get Started with SAS® Visual Data Mining and Machine Learning?

MelodieRush — Tue, 02 Jun 2020 14:30:00 GMT

Slides and Q&A for this session have been added here.

Re: Free Webinar: How Do I Get Started with SAS® Visual Data Mining and Machine Learning?

MelodieRush — Thu, 28 May 2020 20:48:05 GMT

Slides and Q&A will be posted next week once we have complied all the Q&A.

Re: Free Webinar: How Do I Get Started with SAS® Visual Data Mining and Machine Learning?

skalwani — Thu, 28 May 2020 16:39:40 GMT

I went to the spot described but could not find the slides from yesterdays session?

WANTED - Your Insights

AnnaRhyne — Wed, 27 May 2020 13:19:44 GMT

We are regularly asked for stories about how our products have impacted our customers' business. Often they are looking to understand how our software contributes to making peoples lives better, whether that be for their customers or for you in being able to perform your job better, faster, and more efficiently.

Today, you can help provide insights for someone else's purchasing decision. Please take a moment to write a review on Gartner Peer Insights about your experience with SAS.

Reviews are anonymous and take around 10 minutes to complete. As thanks for your approved review, the first 50 reviewers will receive a $25 Visa gift card from Gartner.

Submit your review here.

We appreciate your feedback!

Free Webinar: How Do I Get Started with SAS® Visual Data Mining and Machine Learning?

MelodieRush — Tue, 26 May 2020 14:56:07 GMT

Hi Data Mining and Machine Learning Community,

I’m presenting a live “Ask the Expert” webinar on May 27, 11:00 – 12:00 p.m. ET How Do I Get Started with SAS^® Visual Data Mining and Machine Learning? Please join me.

About the webinar:

SAS Visual Data Mining and Machine Learning is designed for the data scientist, statistician and advanced business analyst. Whether you want to program or point and click, SAS Visual Data Mining and Machine Learning provides innovative algorithms and fast, in-memory processing. This session covers its capabilities and has an accompanying demonstration that provides a view into the components of SAS Visual Data Mining and Machine Learning

What you will learn

Interactive programming in a web-based development environment
Intelligent automation including Automatic Feature Engineering node for automatically cleansing, transforming, and selecting features for models.
Natural language generation
Embedded support for Python & R languages
Deep learning with Python (DLPy)
SAS® procedures (PROCs) & CAS actions
Highly scalable, distributed in-memory analytical processing

Are there any other specific questions you’d like covered? Let me know by responding to this thread.

Want more tips? Be sure to subscribe to the Ask the Expert board to receive follow up Q/A, slides and recordings from other SAS Ask the Expert webinars. To subscribe, select Subscribe from the Options drop down button above the articles.

Can't join the live event? You can view this and other Ask the Experts sessions on-demand here.

Re: Event Based Sampling in Model Studio SAS Viya

ravnen55 — Tue, 26 May 2020 09:28:02 GMT

One related question:

How do you oversample in model studio?

If by oversampling we simply mean duplicate some of the rareevents cases, and we do this before partitioning, then chances are that some of the cases in the train and the test datasets er identical, and that seems to me like a clear data leakage.

But how do you, in model studio, oversample only to the train dataset? Or should you do something else?

SAS enterprise miner decision tree splitting

Scott86 — Mon, 25 May 2020 03:16:37 GMT

Hi Everyone,

I am creating a decision tree and I want it to be only 3 nodes deep. I am controlling the splitting by using the leaf size under the Node area for Decision tree models, I am setting it to 200 (that is a minimum of 200 obs per leaf). The issue is the target I am splitting up which is binary (1 and 2) has a low volume. The target 2 only account for 0.44% of the observations. This is creating difficulty in the tree splitting. When I set the leaf size to 200 it goes to deep but when I increase it the tree does not split at all. I have tried changing the Significance level but it has no affect.

Nominal target Criterion is ProbChisq and significance level is 0.2.

Any help on how to tune the Hyper-Parameters to prune the model would be great.

Thanks

Scott

How to determine the one optimal decision threshold across multiple predictive models

JamieTee — Sun, 24 May 2020 03:10:13 GMT

I am currently wanting to construct predictive models to identify patients who may have a misdiagnosis of a certain disease. However, the issue is that I have multiple databases and I do not want to combine the data into one set, so I need to create predictive models for each database. On top of that, I wanted to use different model approaches and use the most accurate model.

I'm currently facing the problem of determining a way to identify the optimal decision threshold to categorize the results of the predictive model as having been misdiagnosed (p=1) or not having been misdiagnosed (p=0). I've read about how we can use Youden's Index to determine the optimal cutoff for a model, but since I'll have multiple models for each of the database, would it make sense to use the SAME cuttoff across all the models and databases or have one for each database (but keep it consistent for each model within the database)? I'm a bit lost on what the best approach is. I haven't been able to find papers that provide detail on how they determine optimal decision thresholds on multiple models at the same time

P.S. I've read about how you can use the cost of each result (i.e., TP, TN, FP, FN) to determine the threshold, but these values are unknown for my disease of interest.

Sorry about the long post, and thank you for your help in advance!

Your input is needed: Participate in the Rexer Analytics 2020 Data Science Survey

AnnaBrown — Thu, 21 May 2020 15:35:33 GMT

Hello community,

Rexer Analytics has been conducting the Data Science Survey since 2007. Each survey explores the analytic behaviors, views and preferences of data scientists and analytic professionals.

Make sure your perspective is included by taking the 2020 Survey! Your responses are completely confidential. A free report summarizing the findings will be available to everyone 4Q-2020.

Please share with your network of other data analysis professionals who may be interested.

Survey details:

— Link to Start the Survey (It takes about 20 minutes)

— Access Code: A2TF9

It is OK to share this Access Code with others, it can be used by multiple people.

— The Rexer Analytics main Data Science Survey page also has more survey information & FREE downloads of the 2007-2017 Survey Summary Reports

Re: Error when running macro to generate TIME_DIM data

WorkingMan — Sun, 17 May 2020 12:14:48 GMT

Hi, I would wish to copy out the full log but the server that host this SAS environment disabled any form of copy/paste. I typed those error messages out myself. If possible, I can provide screenshot.

Re: Error when running macro to generate TIME_DIM data

PaigeMiller — Sun, 17 May 2020 10:55:53 GMT

You can't just show us the error message. That isn't sufficient. We need to see the ENTIRE log for this macro, code, NOTES, WARNINGS, ERRORs. Please copy the log as text (not as a screen capture) and then click on the </> here and paste the entire log into the window that appears.

Error when running macro to generate TIME_DIM data

WorkingMan — Sun, 17 May 2020 10:06:45 GMT

Hi, in SAS Credit Scoring, there is TIME_DIM table under DIM library. From my understanding, there is a macro script that will load new month and year into this table when we execute. I want to load new years into this table and i managed to find the macro for this.

%bnkfdin;
%bankfdn_create_time_dim(yr_start=2004, yr_end=2016, mapto=);

When I run this in SAS Enterprise Guide, I get this error:

WARNING: Apparent symbolic reference FM_GRAIN not resolved.
ERROR: A character operand was not found in the %EVAL function or %IF condition where xxxxxxxxx
The condition was :
&FM_GRAIN ne 1 and &MapTo ne %sysfunc(dequote(&WEEK_START_FLG)) and &MapTo ne %sysfunc(dequote(&WEEK_END_FLG))

I have been trying to understand and browse through administrator guide for Credit Scoring, I still dont find any clue at all. In admin guide, I found this:

Set the correct value of the FM_GRAIN parameter in the
Custpara.Parameter table. Only numeric values are supported for this
parameter. Here are the values:
n 1 is Month (default value)
24 Chapter 2 / Installing SAS Credit Scoring
n 3 is Day

I have no idea what is custpara.parameter. Why is this SAS CS Admin Guide has such instruction that is so vague?

Is there anyone that can help me with this?

Re: ERROR: Connection failed. Server returned: Session reconnect failed: Could not find the specifie

AnnaBrown — Fri, 15 May 2020 17:59:43 GMT

Fantastic, ! Thanks for sharing your solution with the community.

Anna

Re: ERROR: Connection failed. Server returned: Session reconnect failed: Could not find the specifie

klroesner — Fri, 15 May 2020 17:39:30 GMT

Hello,

We have an answer from the technical support. The CAS action works as designed. The action must always be called with constraints such as minimal support, otherwise it leads to exploding data growth.

Cheers, Klaus

How to make a plot of residuals in survival analysis SAS EM

amsa1996 — Fri, 15 May 2020 09:59:37 GMT

Hi!

I have made a survival analysis in SAS EM using the Survival node (change time format). I want to detect anomalies/prediction errors in SAS EM from the survival analysis. Is it possible for me to make a plot of the residuals from the analysis in SAS EM? If yes, can someone help me with how i do this? If no, can someone help me with other options?

My target variable are categorical (0-1-2).

I know that i can use the export data ellipsis under "General" and then going on plot under "Actions". But then there are no predicted values such as "P_target" for my target variable that i can use.

Best regards,

Re: SAS Enterprise Miner 15.1 and R 3.6.3

JosvanderVelden — Thu, 14 May 2020 17:01:31 GMT

The enterprise miner 15.1 documentation on PMML support is here: https://documentation.sas.com/?docsetId=emref&docsetTarget=n0jswh909h49dnn1mq562smtf4jz.htm&docsetVersion=15.1&locale=en
If you are using the open source integration node the help is here: https://documentation.sas.com/docsetId=emref&docsetTarget=p19968g5knhu81n1llu0u3lkz0ub.htm&docsetVersion=15.1&locale=en#p1m9wwjx4jjn2cn1r3kov6wcu2nf

Does this help?

Re: Enterprise Miner: What is "Selection Depth"?

WendyCzika — Wed, 13 May 2020 20:28:48 GMT

The various assessment measures where you would use Selection Depth are documented here in the chapter on the Model Comparison node:

https://go.documentation.sas.com/?docsetId=emref&docsetVersion=15.1&docsetTarget=p01jgc9rmzsg37n1lfncp67t0unm.htm&locale=en#n06jncp1dkerydn134rb66la3vt2

and the Selection Depth property is described at the bottom of this section (or you can just search the chapter on the Model Comparison node for "selection depth") - essentially it is the decile you want to use for lift and gain-type statistics when comparing your models.

https://go.documentation.sas.com/?docsetId=emref&docsetVersion=15.1&docsetTarget=p01jgc9rmzsg37n1lfncp67t0unm.htm&locale=en#n0dh4rt3dp7bijn1m4m8to5u0063

Possible segment plot issue | RFM analysis | SAS Enterprise Miner

Vassilis — Wed, 13 May 2020 18:03:53 GMT

Hi all,

I am facing the following issue.. I am working on an RFM dataset previously extracted from SAS Enterprise Guide. Using the Cluster node, I keep getting this strange result on the Segment Plot where the Frequency variable shows a uniform distribution across clusters.

I exported the Frequency values for each segment and created respective number of histograms in Enterprise Guide, where it showed that there is not a uniform distribution in the values.

Is it then a bug from Enterprise Miner? Can I trust the analysis ignoring the Visualizer in the Segment Plot?

Thank you in advance for any guidance or information.

Enterprise Miner: What is "Selection Depth"?

PaigeMiller — Wed, 13 May 2020 14:05:55 GMT

As I am following this set of videos describing a modeling example in SAS Enterprise Miner, I see that the videos always use a "selection depth" of 5 to compare models. The documentation from SAS does not explain what "selection depth" is, it just says you can set it to 5 or 10 or some other number. Can someone explain what "selection depth" means (which, by the way, should be in the documentation somewhere) and why you might choose it to be the value of 5 rather than some other value?