Saving an output dataset using Proc genmod and HADOOP/Impala

aellman — Thu, 13 Oct 2016 21:53:41 GMT

Hello,

I have a dataset that is housed in Hadoop HDFS and I would like to perform a proc genmod using this dataset. For instance, my libname is set as

libname dchd odbc dsn='xxx' schema='data' dbcommit=99999 insertbuff=99999 readbuff=99999;

I have created a dataset called 'ds' that is stored in HDFS as data.ds

I now want to perform genmod as follows

ods output Estimates=data.output (keep=event cohort_year Label MeanEstimate MeanLowerCL MeanUpperCL);

proc genmod data=data.ds;

model num_event= /d=p link=log offset=log_years;

by event cohort_year;
estimate " " int 1 /exp ;
ods select modelinfo parameterestimates estimates modelfit;
run;

The model runs fine but when I try to store an output dataset I get this error:

I am running Enterprise Guide 7.12

Thanks.

Further text from log:

ERROR: During insert: [Simba][ImpalaODBC] (110) Error while executing a query in Impala: [HY000] : AnalysisException: Possible loss
of precision for target table 'aellman.OUTPUTH1'. Expression 'cast(2008 as double)' (type: DOUBLE) would need to be cast to
DECIMAL(11,0) for column 'cohort_year'
NOTE: PROCEDURE GENMOD used (Total process time):
real time 7.95 seconds
user cpu time 2.72 seconds
system cpu time 0.17 seconds
memory 10176.15k
OS Memory 34488.00k
Timestamp 10/13/2016 02:50:50 PM
Step Count 75 Switch Count 1864
Page Faults 0
Page Reclaims 0
Page Swaps 0
Voluntary Context Switches 7810
Involuntary Context Switches 43
Block Input Operations 0
Block Output Operations 0

Re: Saving an output dataset using Proc genmod and HADOOP/Impala

ballardw — Thu, 13 Oct 2016 22:53:21 GMT

Did you try sending the output from genmod to a different library than the source data? I'm not sure that the model procedures' ODS OUTPUT speaks HADOOP for output

Re: Saving an output dataset using Proc genmod and HADOOP/Impala

Reeza — Fri, 14 Oct 2016 00:46:01 GMT

Maybe Hadoop is different but the convention is typically libname.memname.

Youre trying to do schemaname.memname

I would try just using the work library for starters though.

Ods output estimates=output;

Usually the ODS SELECT statement also precedes the proc, but not a requirement.

topic Re: Saving an output dataset using Proc genmod and HADOOP/Impala in SAS Procedures

Saving an output dataset using Proc genmod and HADOOP/Impala

Re: Saving an output dataset using Proc genmod and HADOOP/Impala

Re: Saving an output dataset using Proc genmod and HADOOP/Impala