BookmarkSubscribeRSS Feed
moorsd
Obsidian | Level 7

Does anyone know if SAS/ACCESS to Hadoop in 9.4 M3 supports creating partition files in Hive in a non-text format i.e. ORCFile?

 

I've tried to specify creating a table as an ORCFile usign the LIBNAME option in SAS 9.4 M2: 

 

DBCREATE_TABLE_OPTS="stored as ORCFile"

 

However, when I add the data set option to create a partition file: 

 

DBCREATE_TABLE_OPTS="PARTITIONED BY (x_facility_offer_cd VARCHAR(4))"  

 

The resulting HiveQL is showing the table is being created as a TEXTFILE:

 

PARTITIONED BY (x_calling_system_cd VARCHAR(4))  ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' LINES TERMINATED BY '\012' STORED AS TEXTFILE 

 

Also, it seems that indexes on Hive tables can't be created using the SAS/ACCESS to Hadoop in SAS 9.4 M2, is this fixed in M3?

 

The indexing is an issue, as we can't partition ORCFile(which are pre-otimised with an internal index), so we're left with un-optimised TEXTFILE formats.

 

Any help / info would be appreciated.

 

Thanks

 

 

David

1 REPLY 1
LinusH
Tourmaline | Level 20

"In the third maintenance release for SAS 9.4, these features are new or enhanced.

  • You can use the new CONFIG= and CONFIGDIR= LIBNAME and data set options to define the name and location of bulk-load configuration files.
  • Support for DBCREATE_TABLE_OPTS= LIBNAME option is new."
     
    To verify your specific case, you might want to ask SAS tech support to verify that this would work once you upgrade to M3.
Data never sleeps

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

How to connect to databases in SAS Viya

Need to connect to databases in SAS Viya? SAS’ David Ghan shows you two methods – via SAS/ACCESS LIBNAME and SAS Data Connector SASLIBS – in this video.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 1 reply
  • 1612 views
  • 0 likes
  • 2 in conversation