SAS Data Integration Studio, DataFlux Data Management Studio, SAS/ACCESS, SAS Data Loader for Hadoop and others

SAS EG & Hadoop Data Checks

Occasional Learner
Posts: 1

SAS EG & Hadoop Data Checks



I am fairly new to hadoop and using SAS EG to access the data.


I want to run a series of data checks on the data that is stored in hadoop i.e. for a particular table (or library/database) for each (tables)columns identify the min, max, missing, no of records etc...


I tried using the traditional PROC Contents/ PROC Datasets but it takes ages given the volume of data etc..


Is there a better way to run the two commands in hadoop via hive sql?


Effectively I am after a table which shows:


table name, column_name, column type, no of records, no of missing values, no of distinct values, min value, max value, min length, max length,



Super User
Posts: 5,853

Re: SAS EG & Hadoop Data Checks

Proc contents should give the structure, but it shouldn't take much time.
The other stats isn't available in contents nor datasets procedure. For those i sugest that you use SQL. But it requires a full table scan given the nature of your requirement. Just be sure that the SQL is sent to Hive. Try with a small table first, and use
Options msglevel = I sastrace = ',,,d' SASTRACELOC = saslog;
for verification.
Data never sleeps
Ask a Question
Discussion stats
  • 1 reply
  • 2 in conversation