New library articles in SAS Support Communities

Advanced configuration of a Shock in SAS Risk Factor Manager

MihaiViju — Thu, 25 Apr 2024 20:33:07 GMT

Advanced configuration of a Shock in SAS Risk Factor Manager

Wavelet Analysis using SAS/IML: Thresholding the Detail Coefficients to Remove High-Frequency Noise

AriZitin — Fri, 26 Apr 2024 15:18:39 GMT

Recently in the SAS Community Library: SAS' @AriZitin shows you how wavelets can be used to remove unwanted high-frequency noise from digital signals.

Don’t Listen to Ron White. Cluster Profiling is Right!

MarcHuber — Thu, 25 Apr 2024 19:56:48 GMT

Don’t Listen to Ron White. Cluster Profiling is Right!

SAS Viya at SAS Innovate - What’s New in Data & AI

thiago — Wed, 24 Apr 2024 21:22:12 GMT

Want to know what’s new in SAS Viya? In this SAS Viya Release Highlights Show, we’re packing our bags and heading to Las Vegas to showcase the latest SAS Viya announcements. We’ll be covering what’s new when it comes to SAS Viya Workbench, generative AI, copilots and even dive into topics like quantum computing. You’ll hear from SAS experts about how SAS is investing in these exciting technology areas.

Maximize Coding and Data Freedom with SAS, Python and Databricks

larsarne — Wed, 24 Apr 2024 08:49:18 GMT

Freedom of choice is now more important than ever when you want to get the most out of your data and AI specialists. Boasting compatibility with 80+ data sources, including widely adopted open data standards and formats such as Parquet and ORC, the possibilities for data access are virtually limitless with SAS Viya. Check out this blog to look into how this freedom also extends to coding language with SAS Viya - in particular Python.

Why Do I Need SAS Intelligent Decisioning and SAS Model Manager to Achieve Analytics Success?

Crystal_Baker — Tue, 23 Apr 2024 14:27:49 GMT

Watch this Ask the Expert session to learn how the integration between these SAS solutions helps MLOps engineers, data scientists and business users improve analytics success by working together.

Watch the Webinar

You will learn about:

How SAS® Viya® supports enterprise team collaboration to efficiently build AI and analytics systems.
Core functionality of SAS Model Manager for MLOps engineers, including registration, testing, governance, monitoring and integration with solutions such as SAS Intelligent Decisioning.
Core functionality of SAS Intelligent Decisioning for decision analysts, including rule building, ML/AL implementation, code support, orchestration and deployment.

The questions from the Q&A segment held at the end of the webinar are listed below and the slides from the webinar are attached.

Q&A

After model decay is detected, what are the next steps?

We didn't dive too deep into model decay in this example, but when we develop our machine learning models, they are just representations of a pattern that they detect in the world around us at a specific time point. Our world is ever changing, new things appear all the time, things phase out. So, when our models are trained to work with a specific pattern at a single point in time, it becomes less effective at prediction when that pattern is no longer useful. This is a process called model decay.

Within our SAS model management tool, we have capabilities for setting thresholds for sending tasks or notifications whenever your model decay doesn't meet that threshold, so that users can come in and decide what's next. Do they retrain their models in SAS Model Studio? Do they select a challenger model that looks like it's still performing as well as the champion or better? We do support performance monitoring for multiple models so you can share those challenger models side-by-side those champions. You can quickly see it's time to replace my production model with a challenger or decide to retire the project to build new models. There are a few different options that are available, but we do help with letting individuals know when it is time to address model decay. From there, we can replace production models when we go to deploy; there's a toggle to replace the model or replacing the model with the same name. Or we can just let Crystal know that it's time to swap out that model, as you've seen in the demonstration. We can even very quickly share our models. There's a share button, so we can even share it over teams if I want to speed things along.

What else can I do to ensure my decision flows leverage Responsible and Trustworthy AI best practices?

There are a few things that we can do to better make sure that our decisions fall into that responsible and trustworthy AI category. First, when we are using models within our decisions, we want to make sure that we're working with the data scientists or whoever is developing and maintaining those models. Because a large part of how our decisions depend on the outputs those models. So, we need to make sure that our models are up to date. If there needs to be retraining, retrain the models before our models get stale. In addition, we also want our decisions to be transparent and be able to explain how a decision was made. With SAS intelligent decisioning, there's a few ways we can do this. When we publish decisions, navigate to the Deployments tab to manage the different deployed decisions that we have. We can understand where they've been deployed to, who's deployed them, and the date of publishing. Also, with this, we can get a better understanding of how the decision is running and what it is using to make decisions by easily generating a report that helps us create better transparent decisions.

We can also track decisions with decision path analysis and rule-fired analysis. We can also track decisions with decision path analysis and rule-fired analysis and the output decision variables. With this data we can do analysis to better understand how decisions are made and ensure that we are making responsible decisions.

While developing category models and churn models, do you actually code or is it done by SAS Viya itself?

What's nice about SAS Viya is that it's almost like a choose your own adventure when it comes to how you develop your models. It can be yes code, no code, low code, or somewhere in between. For this particular example, these models that we use were developed using a tool called SAS Model Studio, which is a GUI (Graphical User Interface) based approach where you don't have to do any coding yourself. You can drag and drop nodes into a pipeline, you can use automated machine learning to develop that pipeline for you. You don't actually have to code at all to build out these models we used today. But that doesn't mean you can't code. Users can also code in SAS and Python or R to develop their models. We have a few different options for those users. It is nice and flexible so you can start to bring together skill sets across your organization, whether they can code and enjoy it and just want to stick with that method, or if they prefer to iterate very quickly using a drag and drop tool. There is a variety of different options for how these models can be developed.

How can I learn more?

Sophia: Besides joining us at SAS Innovate, both Model Manager and Intelligent Decisioning are very active on SAS communities. I post under the SAS Model Manager label whenever we have new features, and many of our experts in the community post interesting use cases they come across, and interesting problems they've solved. So, it's definitely worthwhile to subscribe to the SAS Model Manager label on SAS Communities.

Crystal: SAS Communities is always a good one for Model Manager or SAS Intelligent Decisioning. We have some blogs on the sas.com page and then also in SAS Communities. I also suggest taking a look at the previous because we have a lot of good tutorials or how to videos that show you how we move throughout either SAS Model Manager and SAS Intelligent Decisioning. The previous webinars can be especially helpful if you are new to either of these solutions.

SAS Model Manager Communities Link: SAS Communities: SAS Model Manager

SAS Intelligent Decisioning Communities Link: SAS Communities: Decisioning

SAS Ask the Expert Webinars: Ask the Expert - SAS Support Communities

Did you use logistic regression to develop the model?

In this example, the category model is a rules-based text model, and the churn model is a gradient boosting model, but logistic regression models and many other machine learning models are supported.

Recommended Resources

Essential Functions of SAS Intelligent Decisioning

Manage Models in SAS Viya Training

SAS Intelligent Decisioning Homepage

SAS Model Manager Homepage

SAS Viya Homepage

Please see additional resources in the attached slide deck.

Want more tips? Be sure to subscribe to the Ask the Expert board to receive follow up Q&A, slides and recordings from other SAS Ask the Expert webinars.

Email Authentication Mechanisms - an Introduction

DietmarBurgfeld — Tue, 23 Apr 2024 09:10:39 GMT

Email Authentication Mechanisms – an Introduction:

Email authentication is a set of techniques and protocols designed to verify the legitimacy of an email message and to prevent email fraud, such as phishing and spoofing. The goal is to ensure that the sender of an email is who they claim to be and that the content of the email has not been tampered with during transit. Email authentication helps in building trust between senders and recipients, reduces the likelihood of spam and phishing attacks, and improves overall email security. Several key authentication mechanisms are commonly used in email:

SPF (Sender Policy Framework): SPF is a framework that allows domain owners to specify which mail servers are authorized to send emails on behalf of their domain. This is achieved by publishing SPF records in the DNS (Domain Name System). Receiving mail servers can then check the SPF records to verify that an incoming email is sent from an authorized server.
DKIM (DomainKeys Identified Mail): DKIM is a method for associating a domain name with an email message, allowing a person, role, or organization to claim some responsibility for the message. It involves the use of cryptographic signatures that are added to the email headers. The receiving mail server can verify the signature using the public key published in the DNS.
DMARC (Domain-based Message Authentication, Reporting, and Conformance): DMARC builds on SPF and DKIM by providing a way for domain owners to set policies for how their emails should be handled if they fail authentication checks. DMARC also includes reporting features that allow domain owners to receive feedback on email authentication failures.
BIMI (Brand Indicators for Message Identification): BIMI is a relatively newer standard that enables organizations to display their brand logos next to authenticated emails in the recipient's inbox. It relies on the combination of DMARC and a verified logo file that is referenced in the DNS.

By implementing these authentication mechanisms, email senders can demonstrate the legitimacy of their messages, and recipients can have greater confidence that the emails they receive are not forged or manipulated. Additionally, DMARC policies help prevent attackers from abusing a brand's domain for malicious purposes. Overall, email authentication is a crucial aspect of modern email security and is widely used to combat various forms of email fraud.

--------------------------------------------------------------------------------------------------------------------------------------

SPF:

Sender Policy Framework (SPF) is an email authentication method designed to detect forged sender addresses during the delivery of the email. It is part of the broader set of technologies aimed at preventing email spoofing and phishing. SPF allows the owner of a domain to specify which mail servers are authorized to send emails on behalf of that domain.

How SPF works:

DNS Records: The domain owner publishes SPF information in the Domain Name System (DNS). SPF records are TXT records that contain information about the authorized mail servers for that domain.
Receiving Mail Server Checks SPF Record: When an email is received, the receiving mail server can check the SPF record of the sender's domain by querying the DNS for the SPF TXT record.
Validation: The receiving mail server compares the IP address of the sending server against the list of authorized IP addresses specified in the SPF record. If the IP address matches one of the authorized servers, the email passes the SPF check.
Handling Failures: If the SPF check fails, the receiving mail server can take various actions based on the SPF policy specified by the domain owner. This might include marking the email as suspicious, moving it to the spam folder, or even rejecting it outright.

SPF records use a simple syntax to specify which IP addresses are allowed to send emails on behalf of a domain. For example, a basic SPF record might look like this:

In this example:

v=spf1 indicates the SPF version.
ip4:192.168.1.1 allows the specific IP address 192.168.1.1 to send emails.
include:spf.example.com includes the SPF record of another domain (spf.example.com) in the check.
-all signifies a strict policy, indicating that if the IP address is not listed in the SPF record, the email should be treated as unauthorized.

SPF is just one component of email authentication. Other techniques like DKIM (DomainKeys Identified Mail) and DMARC (Domain-based Message Authentication, Reporting, and Conformance) are often used in conjunction with SPF to enhance email security. These mechanisms collectively help in reducing spam, phishing, and email spoofing.

DKIM

DKIM functions as a cryptographic authentication method. It allows email receivers to verify that an email claiming to have come from a specific domain was indeed authorized by the owner of that domain. This verification process occurs by validating the cryptographic signature included with the email.

How DKIM works:

Message Signing: When an email is sent, the sending mail server adds a unique DKIM signature header to the message. This signature is created using a private cryptographic key associated with the sending domain.

Public Key Retrieval: The receiving mail server, upon receiving the email, retrieves the public key from the DNS records of the sender's domain.
Signature Verification: Using the public key, the receiving server verifies the DKIM signature attached to the email. If the signature is valid and the message has not been altered in transit, the email is deemed authentic.

Benefits of DKIM

Enhanced Email Deliverability: DKIM-signed emails are less likely to be marked as spam or phishing attempts by email filters, thus improving deliverability rates.
Mitigation of Spoofing and Phishing: By verifying the authenticity of the sender, DKIM helps in reducing the risk of email spoofing and phishing attacks.
Builds Trust with Recipients: Implementing DKIM instills confidence in recipients that the emails they receive are indeed from legitimate senders, fostering trust and credibility.
Compliance Requirements: DKIM compliance is often a prerequisite for adhering to various industry regulations and standards, such as GDPR (General Data Protection Regulation) and HIPAA (Health Insurance Portability and Accountability Act).

DMARC

DMARC, which stands for Domain-based Message Authentication, Reporting, and Conformance, is an email authentication protocol and reporting system designed to combat email phishing and fraud. DMARC builds on the existing email authentication mechanisms SPF (Sender Policy Framework) and DKIM (DomainKeys Identified Mail) by providing a way for domain owners to specify how email failures should be handled and by offering reporting capabilities.

Key components of DMARC:

Authentication Mechanisms:

SPF (Sender Policy Framework): Verifies that the sending mail server is authorized by the domain owner.
DKIM (DomainKeys Identified Mail): Ensures that the email content has not been tampered with and is signed by the private key associated with the sending domain.

DMARC Policies:

Domain owners publish DMARC policies in their DNS records. These policies instruct receiving mail servers on how to handle emails that fail SPF and/or DKIM checks.
DMARC policies include:
- p=none (Monitoring Mode): The domain owner is in "monitoring" mode, and no specific action is taken when authentication fails. However, detailed reports are generated and sent to the domain owner.
- p=quarantine: The email should be treated with suspicion. Receiving mail servers may choose to deliver the email to the recipient's spam or quarantine folder.
- p=reject: The email has failed authentication, and the receiving mail server should reject it outright. The strictest setting.

Reporting:

DMARC provides detailed reports (DMARC reports or aggregate reports) to domain owners about email authentication activity. These reports include information on emails that passed or failed authentication, including the sending IP addresses, SPF results, DKIM results, and more.
These reports enable domain owners to monitor and analyze email authentication activity, identify potential issues, and take corrective actions.

DMARC helps prevent email spoofing and phishing attacks by providing a mechanism for domain owners to assert control over their domains and protect their brand reputation. It also allows them to receive valuable feedback on email authentication failures, helping to improve the overall security of their email infrastructure. Many large email providers and organizations have adopted DMARC as part of their email security strategy.

BIMI

BIMI, which stands for Brand Indicators for Message Identification, is an email authentication standard that aims to enhance email security and brand visibility. BIMI allows organizations to display their brand logos next to authenticated emails in the recipient's inbox. This visual indicator helps users quickly recognize legitimate emails from trusted senders and builds trust in the email communication.

Key components of BIMI:

DMARC Authentication:

BIMI relies on the DMARC (Domain-based Message Authentication, Reporting, and Conformance) protocol for email authentication. DMARC helps ensure that the sender's domain is legitimate and prevents email spoofing and phishing attacks.

Verified Mark Certificates (VMC):

To participate in BIMI, organizations need to obtain a Verified Mark Certificate (VMC). A VMC is a digital certificate that confirms the association between the sender's domain, the brand logo, and the DMARC authentication. It acts as a proof of authenticity for the brand logo.

DNS Records:

BIMI information is published in the DNS (Domain Name System) records of the sender's domain. This includes the location of the brand logo and the associated VMC.

Display of Brand Logo:

When an email from a BIMI-enabled sender passes DMARC authentication, and the recipient's email client supports BIMI, the brand logo is displayed next to the email in the recipient's inbox. This provides a visual confirmation of the sender's authenticity.

The benefits of BIMI include:

Brand Trust: The presence of a recognizable brand logo next to an email can enhance trust and help users distinguish legitimate emails from potential phishing attempts.
Improved Email Deliverability: BIMI is an additional incentive for organizations to implement DMARC authentication. Email providers may prioritize the delivery of authenticated emails, contributing to better deliverability rates.
Reduced Phishing Risks: By visually verifying the authenticity of the sender, BIMI helps users identify phishing attempts more easily, reducing the risk of falling victim to fraudulent emails.

BIMI is an emerging standard, and its adoption is gradually increasing. As more organizations implement DMARC and obtain VMCs, BIMI has the potential to become a valuable tool in the fight against email fraud while providing a better user experience in email communication.

Comparing Alerting Capabilities of Alertmanager with Grafana

AjmalFarzam — Mon, 22 Apr 2024 20:12:55 GMT

Comparing Alerting Capabilities of Alertmanager with Grafana

Auditing user sessions in Viya 3.5 (update)

AjmalFarzam — Mon, 22 Apr 2024 20:10:25 GMT

Auditing user sessions in Viya 3.5 (update)

Solving Linear Regression Models via Mathematical Programming: Ordinary Least Squares

ChanceTGardener — Mon, 22 Apr 2024 13:58:53 GMT

Solving Linear Regression Models via Mathematical Programming: Ordinary Least Squares

Efficiently Modeling Interval Targets Using Bayesian Additive Regression Trees -Part 2

smanoj — Mon, 22 Apr 2024 09:36:38 GMT

Efficiently Modeling Interval Targets Using Bayesian Additive Regression Trees -Part 2

SAS Viya CLI container image in Azure DevOps (March 2024 Update)

DavidEstreich — Mon, 22 Apr 2024 08:33:35 GMT

SAS Viya CLI container image in Azure DevOps (March 2024 Update)

SAS Visual Investigator: It's Not JUST for Fraud

BethEbersole — Fri, 26 Apr 2024 15:04:36 GMT

Recently in the SAS Community Library: SAS' @BethEbersole reveals 4 steps to stop money laundering, solve law-enforcement cases, find missing children and more with SAS Visual Investigator.

An Overview of Tripwires in SAS Law Enforcement Intelligence

RhondaWilliams — Fri, 19 Apr 2024 13:54:42 GMT

An Overview of Tripwires in SAS Law Enforcement Intelligence

Everyone knows that in today’s world, we can use technology to our advantage in our everyday lives. Let’s say you need a last-minute birthday gift for your spouse. You go online, order, click and pay. Then you anxiously keep your phone next to you waiting for that ‘ding’ sound that you have a notification. DING! You can see the order has been processed. DING! The order has been packed. DING! A shipment label has been created. DING! Your package has been received by the shipping service. DING! The shipment is on its way. DING! The shipment is out for delivery. Technology is helping you keep a close eye on that package. You might even frantically check your location app and home cameras to see where your spouse is. You get the point. Since you are having it delivered to the house you want to get to the gift before your spouse does. DING! A cat walked into the view of your doorbell camera. UGH. It’s not what you need so you ignore it. DING! It finally happens...

Warning: Risk of Serious Harm!

RhondaWilliams — Fri, 19 Apr 2024 13:55:29 GMT

Warning: Risk of Serious Harm!

Hindsight is 20/20. Haven’t we all heard that phrase before? I know I have. That’s something I said myself just a few years ago. It was November, the sun was starting to set, and the water was freezing cold. While exploring the shoreline in Myrtle Beach, South Carolina, with my family, I had fallen a little behind. At some point, I hurried on in an attempt to catch up. I was fully dressed in every-day clothes and had no plans of getting in the water. Every step I took was meticulously calculated to avoid getting wet. In fact, I was quite cautious as I was very aware that I had my shiny brand-new cellphone in my pocket because I had been capturing precious moments of my family together. Then it happened. No warning...

SAS Innovate - SAS Studio Git Integration Links

StephenFoerster — Thu, 18 Apr 2024 02:00:46 GMT

If you attended the SAS Innovate session on SAS Studio's Git integration capabilities, you may be interested in the links that were shown. Here they are.

Data to Databricks? No need to recode - get your existing SAS jobs to SAS Viya in the cloud!

larsarne — Wed, 24 Apr 2024 14:57:48 GMT

What if you could make your valuable and business-critical SAS Data Integration jobs work with Databricks in the cloud in a fraction of the time it would take to recode the jobs in another language or system? You can! Read this post to see how.

Exploring the Exciting World of AI: Insights from TRANSFORM in Berlin

SarahMueller — Thu, 18 Apr 2024 09:53:54 GMT

Last month, I ventured into the heart of Berlin to attend TRANSFORM—an event dedicated to unravelling the mysteries of digital transformation, with a special focus on the remarkable world of artificial intelligence (AI). Let me take you through the journey, where innovation met inspiration at every turn.

TRANSFORM wasn't just any gathering; it was a buzzing hub of executives, decision-makers, and digital experts from various industries, all coming together to explore the ever-evolving landscape of technology-driven business revolution. Amidst the discussions and showcases, one theme stood out: the transformative power of AI.

At TRANSFORM, we heard how AI isn't just a far-off idea anymore; it's already changing how businesses operate worldwide. Companies are using AI to do things like analyze data faster than ever before, create chatbots that help customers, and even predict trends in different industries like healthcare and finance.

But here's the thing: while everyone talks about how important it is to change with the times, not many are actually doing it. Many people are still cautious and waiting to see what happens next. But the truth is, the time to transform is now!

Even though our company, SAS, wasn't on the big stage at TRANSFORM, we're still a big player in the AI world. We're recognized as leaders by Forrester Magazine, and we're helping hundreds of businesses use AI to make better decisions and serve their customers better.

When I came back from the event, I found an article by Forrester that emphasized our role in shaping the future of AI. It was reassuring to see that our efforts were recognized, even we did get a spotlight on the event.

My wrap up

AI is changing the world, and we're part of that change. We might not be the ones standing on stage at big events, but that doesn't mean we're not making a difference. Whether we're recognized or not, our work is vital in shaping the future of technology and business.

As we move forward, let's remember that the journey into the future of AI is exciting and full of possibilities.

We're in the game and we're here to stay.

How Do I Clean My Data Using SAS Programming? Q&A, Slides, and On-Demand Recording

JackieJ_SAS — Tue, 16 Apr 2024 20:50:06 GMT

Watch this Ask the Expert session to learn programming tips and tricks for cleaning your data using SAS programming.

Watch the Webinar

You will learn about:

Issues to consider when cleaning character and numeric data.
Using basic SAS procedures to clean data, such as PROC FREQ, PROC MEANS, PROC UNIVARIATE and PROC PRINT.
Advanced techniques for cleaning data, including downloadable macros.

The questions from the Q&A segment held at the end of the webinar are listed below and the slides from the webinar are attached.

Q&A

Does data cleaning include the temporary files created by SAS?

I think of data cleaning as the process used to investigate values in permanent datasets. You often will create temporary files in that process and will inspect those as well.

How do you normalize observations that include street address?

There is a great chapter in Ron Cody’s book about this. See Chapter 3 on Standardizing Data in Cody’s Data Cleaning Techniques Using SAS.

I often run a proc contents before cleaning steps.

I assumed existing knowledge of the dataset variables in the webinar demonstrations. I agree- a necessary first step when cleaning an unfamiliar dataset is to run PROC CONTENTS to understand the variable names, types, and any formats.

I see Gender has a record with "f" lowercase!

Yes, exactly. Expected values of Gender in the demonstration dataset are “M” or “F”. The lowercase “f” is a data error that should be addressed.

Why is there a space in "Patient " and "patient number " etc.?

SAS is writing the PUT statement text exactly as printed within the quotation marks. There is a space after “Patient” and “patient number” so that there will be a space between the text and the printed variable value that comes next in the sentence.

Please recommend additionally helpful SAS certification for a clinical trial programmer using SAS 9.4, other than "certification of SAS Certified Professional using SAS 9.4”, and please recommend the study material and books for SAS clinical trial programmers.

SAS released a new version of the Clinical Trials Programmer Professional certification in 2023. The practice materials for this exam list several excellent resources for SAS clinical trial programmers.

Can notdigit function check numbers like 9.83 or there is separate function for that?

Note that the NOTDIGIT function is applied to a character variable only. It cannot be applied to a numeric variable. If you applied the NOTDIGIT function to the character string “9.83”, it would return the position of the decimal, which is 2. I assume from your question, that you would like a function to return zero for a string that includes only numbers or a decimal point.

Switch to the FINDC function in this case. By default, this searches a character string for the first position of any character in a provided list of characters. When you add the “K” modifier, you can search for the first position of a character that is NOT in the list of characters provided. The expression FINDC(“9.83”, “0123456789.”, “K”) looks for the position of the first character that is not a number or decimal point. Since “9.83” contains only numbers and a decimal point, the expression returns 0.

Why does $Gender_Check format not work for missing records in proc freq? How do we make it work?

The formatted value “Missing” in the $Gender_Check format is not printed, since the default behavior of PROC FREQ is to exclude missing values from the table. To include missing values in the table, add the MISSING option to the TABLES statement. For example: TABLES GENDER / MISSING.

Can you use ODS to change the output destination for the file print statement to someplace other than the results viewer?

You can send the output from the FILE PRINT statement to ODS destinations by including the appropriate ODS statements around the DATA _NULL_ step. For example- adding ODS RTF and ODS RTF CLOSE around the DATA _NULL_ step will send the output to a Microsoft Word file.

Is there a reference document with these commands that can be accessed by webinar participants?

Demonstrations in this webinar were taken from the book Cody’s Data Cleaning Techniques Using SAS by Ron Cody. The author’s SAS Press web page includes a link to download all the code in the book for free.

Can you ask SAS to run the top 10 (or other n) highest and lowest values in proc univariate, instead of top/lowest 5?

Yes, you can use the NEXTROBS option on the PROC UNIVARATE statement to customize the number of extreme values printed. This is covered later in the webinar.

Also, please recommend SAS study materials and books that cover CDISC standard integration.

The best search for SAS Press books is now on Redshelf. I recommend you put CDISC into the “Search for Titles” field. The newest book I recommend is An Introduction to Creating Standardized Clinical Trial Data with SAS by Case and Tian. Another favorite of mine is Implementing CDISC Using SAS by Holland and Shostak.

It is a question related to the output result for Gender variable. Can we show the results in order for invalid values to be shown first and the missing values appear next, etc.; OR is this shown based on the patient ID number ascending?

In places where the code uses a PUT statement within a DATA step, notes about invalid data are printed to the screen in the order that observations are processed in the input dataset. You could sort the input dataset by patient number or another variable to customize the order of the output.

When using a conditional 2nd set statement - does that Concatenate the 2nd DS brought in or interleave it w/the 1st DS?

See the response to the next question.

What does "set Tmp" do in the statement if _n_=1 then set Tmp?

In this example, we applied two SET statements. The first read in the PATIENTS dataset via the statement: SET PATIENTS. The second SET statement conditionally read in the TMP dataset via the statement: IF _N_=1 THEN SET TMP.

The automatic variable _N_ counts the observation number in a SAS dataset. On the first iteration of the DATA step, _N_ is set to 1, so the condition IF _N_=1 is true, and the TMP dataset is read with the second SET statement. The dataset TMP has one observation and two variables. The values of these two variables are brought into the program data vector (memory used in DATA step processing). On the next iteration of the DATA step, _N_=2, so the condition IF _N_=1 is false, and the TMP dataset is not read in again. The data values from the TMP dataset are retained and not set back to missing, since no additional rows are read. Because of this, the data values in the TMP dataset are retained for all observations in the dataset. The variables coming in from the TMP dataset have the same value for all observations, so are constants.

My programmers claim the techniques being used here (e.g., Proc Print) are 'old fashioned'. What do you think?

I think printing a dataset to look at it will never be old fashioned- ha! Certainly, many advanced programming techniques were not covered, for example, PROC SQL and more sophisticated statistical techniques. This webinar also assumes the programmer does want to look at individual observations in a dataset. Automated data cleaning techniques that do not require visual inspection are also available.

For someone that is completely new to SAS, will the book be good for beginners?

For someone completely new to SAS, I recommend Learning SAS by Example: A Programmer’s Guide, by Ron Cody. This book goes through the basics of SAS programming syntax, which is not covered in Ron Cody’s book Data Cleaning Techniques Using SAS.

If I have 2 million rows, can I still use these procedures to clean the data in SAS?

Mostly likely, no. These techniques assume the programmer has enough time to inspect individual observations in a dataset and has a desire to edit individual data values if errors are found. For large datasets, neither of these is usually true. Data cleaning is often automated when analyzing large datasets, which is to say that dataset-wide rules are applied without looking at the data. For example, you might collapse small categories of a categorical variable into one category, truncate outliers, and replace missing data with the variable’s mean or median, all without looking at individual observations.

Many times, I feel I forget functions and can't remember them all. Also, I can see many new keywords used here. How to remember them all and use at work?

I too don’t remember many function names. I often consult the list of functions in the SAS documentation or use Google to find documentation for a particular SAS procedure when I cannot remember the statement syntax. Make sure to use the SAS documentation! Much of the SAS documentation can also be output as a PDF document, which you can then download and search. Look for the PDF link in the upper right of many SAS documentation pages.

Why did you not cross the formatted values with the original variable to see which values fell into which category?

We applied a format to tabulate values of variable so it could take only three possible values – Valid, Missing, or Error. You could cross-tabulate these formatted values by the original unformatted variable to see which values of the original variable were the errors. The FREQ procedure does allow you to cross-tabulate a variable by itself (for example, TABLES GENDER*GENDER), but its FORMAT statement does not allow you to apply a format to only one of the two variable references. In order to cross-tabulate formatted and unformatted versions of a variable, you would need to create a new variable with the formatted values, then cross-tabulate it with the original unformatted variable. Creating a new variable was beyond the scope of this example.

Recommended Resources

Cody’s Data Cleaning Techniques Using SAS on RedShelf

Cody’s Data Cleaning Techniques Using SAS on Amazon

Ron Cody SAS Author Page

SAS OnDemand for Academics Software for Students, Educators, and Independent Learners

Please see additional resources in the attached slide deck.

Want more tips? Be sure to subscribe to the Ask the Expert board to receive follow up Q&A, slides and recordings from other SAS Ask the Expert webinars.

How to understand Chaos Theory Part 2: Using Logistic Map to visualize a Bifurcation

DeeMckoy — Fri, 19 Apr 2024 13:38:48 GMT

How to understand Chaos Theory Part 2: Using Logistic Map to visualize a Bifurcation