Help using Base SAS procedures

Saving an output dataset using Proc genmod and HADOOP/Impala

Reply
Occasional Contributor
Posts: 6

Saving an output dataset using Proc genmod and HADOOP/Impala

Hello,

 

I have a dataset that is housed in Hadoop HDFS and I would like to perform a proc genmod using this dataset.  For instance, my libname is set as 

libname dchd odbc dsn='xxx' schema='data' dbcommit=99999 insertbuff=99999 readbuff=99999;

 

I have created a dataset called 'ds' that is stored in HDFS as data.ds

 

I now want to perform genmod as follows

 

ods output Estimates=data.output  (keep=event cohort_year Label MeanEstimate MeanLowerCL MeanUpperCL);

 

proc genmod data=data.ds;

model num_event= /d=p link=log offset=log_years;

by event cohort_year;
estimate " " int 1 /exp ;
ods select modelinfo parameterestimates estimates modelfit;
run;

 

The model runs fine but when I try to store an output dataset I get this error:

 

ERROR: During insert: [Simba][ImpalaODBC] (110) Error while executing a query in Impala: [HY000] : AnalysisException: Possible loss
of precision for target table 'aellman.OUTPUTH1'. Expression 'cast(2008 as double)' (type: DOUBLE) would need to be cast to
DECIMAL(11,0) for column 'cohort_year'

 

I am running Enterprise Guide 7.12

 

Thanks.

 

Further text from log:


ERROR: During insert: [Simba][ImpalaODBC] (110) Error while executing a query in Impala: [HY000] : AnalysisException: Possible loss
of precision for target table 'aellman.OUTPUTH1'. Expression 'cast(2008 as double)' (type: DOUBLE) would need to be cast to
DECIMAL(11,0) for column 'cohort_year'
NOTE: PROCEDURE GENMOD used (Total process time):
real time 7.95 seconds
user cpu time 2.72 seconds
system cpu time 0.17 seconds
memory 10176.15k
OS Memory 34488.00k
Timestamp 10/13/2016 02:50:50 PM
Step Count 75 Switch Count 1864
Page Faults 0
Page Reclaims 0
Page Swaps 0
Voluntary Context Switches 7810
Involuntary Context Switches 43
Block Input Operations 0
Block Output Operations 0

 

Super User
Posts: 10,532

Re: Saving an output dataset using Proc genmod and HADOOP/Impala

Did you try sending the output from genmod to a different library than the source data? I'm not sure that the model procedures' ODS OUTPUT speaks HADOOP for output

Super User
Posts: 17,899

Re: Saving an output dataset using Proc genmod and HADOOP/Impala

Maybe Hadoop is different but the convention is typically libname.memname. 

 

Youre trying to do schemaname.memname 

 

I would try just using the work library for starters though. 

 

Ods output estimates=output;

 

Usually the ODS SELECT statement also precedes the proc, but not a requirement. 

Ask a Question
Discussion stats
  • 2 replies
  • 250 views
  • 0 likes
  • 3 in conversation