topic getting error when using SPDE ACCELWHERE=YES in libname in SAS Data Management

getting error when using SPDE ACCELWHERE=YES in libname

alandool — Wed, 29 Mar 2017 18:33:04 GMT

Hi ALL,

Getting acquainted with SPDE. I have data sitting in a Hadoop cluster. I have no problem reading into our SAS Grid until I use ACCELWHERE=YES in the libname as shown below. Then I get a "generic error." I am attempting that data subsetting be performed in the Hadoop cluster. Hopefully, it will bring the data back much quicker.

I can remove the ACCELWHERE=YES in the libname and it will run, but it runs for much too long a period although it eventually completes. Code below is just a snippet. (SAS Enterprise Guide version: 7.1. Platform version: 9.4.3.0)

Appreciate any help or advice . . .

Alan

libname hdp_spd spde '/hdfs/vs1/rptg/phi' hdfshost=default PARALLELWRITE=YES PARALLELREAD=YES ACCELWHERE=YES ;

code ...

from hdp_spd.ltv_201702_copy
where INCRD_YR_MN between &bgn_exp and &end_exp ;
ERROR: Generic error.

Re: getting error when using SPDE ACCELWHERE=YES in libname

LouGalway_sas — Tue, 25 Apr 2017 15:12:50 GMT

Hello,

Have you seen this documentation?

http://support.sas.com/documentation/cdl/en/spdsug/67358/HTML/default/viewer.htm#p0u8zw2umzaz5zn17uto6wh9xob4.htm

Specifically, does your SPD Server have an enabled WHERE?

The ACCELWHERE= table option enables a user to override the default state of an undeclared WHERE processing optimization (HADOOPACCELWH=) setting. When SPD Server is using a Hadoop environment, and the SPD Server WHERE processing optimization feature state is not defined in the SPD Server parameter file, the default configuration reverts to NO WHERE processing optimization. You can override an undeclared HADOOPACCELWH= setting by issuing a statement that contains the ACCELWHERE= table option set to YES.

If your SPD Server administrator has enabled WHERE processing optimization via the HADOOPACCELWH= setting, you can use the ACCELWHERE= table option to override and suppress the Hadoop WHERE processing optimization. To override and suppress a HADOOPACCELWH=YES setting, issue a statement that contains the ACCELWHERE= table option set to NO.

Re: getting error when using SPDE ACCELWHERE=YES in libname

alandool — Wed, 26 Apr 2017 18:26:42 GMT

Hi LouGalway_sas,

Thanks for input. Not sure of all our setup details, but we aren't using the SPD Server.

Alan