topic Re: Duplicate in SAS dataset. how about normatilization? in SAS Programming

Duplicate in SAS dataset. how about normatilization?

seamoh — Mon, 08 Jun 2020 19:36:47 GMT

Hi all the members in the community, hope the best week for all of you. I have a question. why there are lots of duplication in SAS datasets(excel and ...). why we don't use Normalization rules before beginning gathering the data?

Thank you very much.

Re: Duplicate in SAS dataset. how about normatilization?

Reeza — Mon, 08 Jun 2020 19:40:58 GMT

Not sure what you mean by duplicates, but re lack of 'normalization' it's typically because SAS is originally Statistical Analysis Software, which takes a different view of data and normalized data isn't usually good for modelling.

Edited.

Re: Duplicate in SAS dataset. how about normatilization?

smantha — Mon, 08 Jun 2020 19:52:08 GMT

Base SAS is not a database in the strictest sense. However it comes close to replicating one. Hence the same rules cannot be applied such as integration check on columns such as column being not null etc. This is in addition to Reeza's answer that SAS is primarily for modeling data and has a different take on what the definition of repeated records mean.

Re: Duplicate in SAS dataset. how about normatilization?

SASKiwi — Mon, 08 Jun 2020 19:53:11 GMT

Normalisation is a concept commonly applied to relational database design to enforce data relationships, and optimise disk storage and database transactional performance. It doesn't have the same relevance for analytical data.

Re: Duplicate in SAS dataset. how about normatilization?

Reeza — Mon, 08 Jun 2020 22:50:46 GMT

I'll just add on to this, DI Studio is designed to support pipelines and normalization, and handle more of the data management side. SAS/BASE/Studio/Foundation are not the same tools but they can do it regardless.

It's more of a build your own though compared to using an Out of the Box tool.

Re: Duplicate in SAS dataset. how about normatilization?

seamoh — Wed, 10 Jun 2020 01:08:01 GMT

I have been in SAS area for one month . you and other friends definitely have more experience than me . I was practicing SAS Essential report 1 and I did an exercise and there was unnecessary duplicate in Report.Of course there are lots of syntax could remove duplication in the reports

then I have this question in my mind.

1. Is it better to design good and normalized datasets for gathering the data from the scratch?

2. Do Data mining and Feature engineering have role in Data analysis in SAS or either in Pyton or R?

I hope I can tell what is in my mind, then you as expert clarify more about.

Thank you very much,

I am looking forward for your answers.

Deeply appreciated

Re: Duplicate in SAS dataset. how about normatilization?

PGStats — Wed, 10 Jun 2020 03:58:12 GMT

You will notice that the vast majority of SAS analytical procedures expect a single dataset as input. A strategy I have used often is to keep the data in (quasi-) normalized tables and to define SQL views to expand the data on the fly for analysis. For non trivial datasets, this is the only robust way to ensure data integrity.

Re: Duplicate in SAS dataset. how about normatilization?

seamoh — Wed, 10 Jun 2020 08:32:47 GMT

Hello and thank you very much for reply. it sounds interesting. would you please introduce me a tutorial described your method or would you please make a tutorial about that. it would help a lot.

Thank you again.

Re: Duplicate in SAS dataset. how about normatilization?

PGStats — Wed, 10 Jun 2020 17:18:09 GMT

Find documentation about SAS/SQL views here:

https://documentation.sas.com/?docsetId=sqlproc&docsetTarget=n0nolnbokay91in1gouzgw3xzl5e.htm&docsetVersion=9.4&locale=en , and here:

https://documentation.sas.com/?docsetId=sqlproc&docsetTarget=p1c2yap6eiwukin18hid2r3h2n4h.htm&docsetVersion=9.4&locale=en#p0rvte35ms3jvan1so5061kwhxhr