BookmarkSubscribeRSS Feed
ShikhaAgarwal
Calcite | Level 5

So the requirement is not just profiling and exploring the data but to be able to highlight the incorrect values by referencing a metadata table which has pattern , datatype, length everything mentioned for each field.

I could find any way to do it in dataflux other than writing my own SAS codes.

Let me know the feasibility please'.

 

3 REPLIES 3
RonAgresta
SAS Employee

Have you tried to author custom metrics and/or business rules and add these to the profile job? Check out the documentation here. The expression language that sits behind the business rule builder is very flexible and it give you some options to meet your use case needs.

Ron

ShikhaAgarwal
Calcite | Level 5

Hi RonAgresta,

Thanks for reverting. I checked the mentioned business rule functionality but i failed to understand how SAS pattern outcomes can be compared to the patterns i have.

Like if i have a table with fields name and the pattern they should follow.

Example:

Fieldname,format

Address Australian postcode,{NNNN}

where N is a number

 

Now i fail to understand how to do the profiling referening this table. I want to highlight those records in the incoming csv files which do not follow the above pattern. Now the roadblocks are:

1)SAS profiling patterns analysis gives outcomes in A and 9 which will not follow the rules which i will create on the business rule manager

2) if in data job i can run some sas codes to bring these pattern annotations in one version , i can do some comparisons and create flags.

3) I could not find how to profile date fields to understand whether its DDMMYY or MMDDYY etc format. I have to give everything in string to get the pattern analysis outcomes. But then it give 99/99/99 . How does it help if i need to parse and create more rules to identify the data format?

 

Appreciate more help in these please.

 

Regards,

Shikha

 

VincentRejany
SAS Employee

Hi Shikha

 

Within SAS DataFlux Data Management Studio, you can create a custom metric to be used within a profile job. Typically for your SSN

 

VincentRejany_1-1590424726717.png

VincentRejany_3-1590424877736.png

VincentRejany_4-1590424891088.pngVincentRejany_5-1590424917769.png

VincentRejany_6-1590424955445.pngVincentRejany_7-1590424973289.png

VincentRejany_9-1590425057816.png

You can also use regular expressions for checking dates.

 

 

 

 

sas-innovate-2024.png

Don't miss out on SAS Innovate - Register now for the FREE Livestream!

Can't make it to Vegas? No problem! Watch our general sessions LIVE or on-demand starting April 17th. Hear from SAS execs, best-selling author Adam Grant, Hot Ones host Sean Evans, top tech journalist Kara Swisher, AI expert Cassie Kozyrkov, and the mind-blowing dance crew iLuminate! Plus, get access to over 20 breakout sessions.

 

Register now!

How to connect to databases in SAS Viya

Need to connect to databases in SAS Viya? SAS’ David Ghan shows you two methods – via SAS/ACCESS LIBNAME and SAS Data Connector SASLIBS – in this video.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 3 replies
  • 580 views
  • 0 likes
  • 3 in conversation