SAS Data Integration Studio, DataFlux Data Management Studio, SAS/ACCESS, SAS Data Loader for Hadoop and others

SAS EG & Hadoop Data Checks

Reply
Occasional Learner
Posts: 1

SAS EG & Hadoop Data Checks

Hi,

 

I am fairly new to hadoop and using SAS EG to access the data.

 

I want to run a series of data checks on the data that is stored in hadoop i.e. for a particular table (or library/database) for each (tables)columns identify the min, max, missing, no of records etc...

 

I tried using the traditional PROC Contents/ PROC Datasets but it takes ages given the volume of data etc..

 

Is there a better way to run the two commands in hadoop via hive sql?

 

Effectively I am after a table which shows:

 

table name, column_name, column type, no of records, no of missing values, no of distinct values, min value, max value, min length, max length,

 

Regards

Super User
Posts: 5,429

Re: SAS EG & Hadoop Data Checks

Proc contents should give the structure, but it shouldn't take much time.
The other stats isn't available in contents nor datasets procedure. For those i sugest that you use SQL. But it requires a full table scan given the nature of your requirement. Just be sure that the SQL is sent to Hive. Try with a small table first, and use
Options msglevel = I sastrace = ',,,d' SASTRACELOC = saslog;
for verification.
Data never sleeps
Ask a Question
Discussion stats
  • 1 reply
  • 207 views
  • 0 likes
  • 2 in conversation