BookmarkSubscribeRSS Feed
☑ This topic is solved. Need further help from the community? Please sign in and ask a new question.
PRAVIN_JAIN
Calcite | Level 5

We use below option while querying in impala.

set PARQUET_FALLBACK_SCHEMA_RESOLUTION=name;

SELECT * FROM db_name.tablename

 

I want to extract same table using SAS. However, unable to set this option 'PARQUET_FALLBACK_SCHEMA_RESOLUTION=name'  in pass through SQL.  Without this option impala looks for column order and wrong values are being displayed. 

 

Please help how this option can be set in Pass through SQL. I tried in '&impala.' connection string and as below statement but it doesn't work - 

execute (set PARQUET_FALLBACK_SCHEMA_RESOLUTION=name) by impala;

 

sample code - 

proc sql;
connect to impala (&impala.);
execute (CREATE VIEW IF NOT EXISTS &BASETABLE. as
(SELECT * FROM db_name.tablename )) by impala;
disconnect from impala;
quit;

1 ACCEPTED SOLUTION

Accepted Solutions
Tom
Super User Tom
Super User

Are you saying the Impala cannot figure out how to names the variables it reads from a parquet file?

 

Can you try setting that option as part of the connection string? Will it stay in effect for your whole session once set?

 

Did you try issuing the SET statement with EXECUTE() before issuing the query with CONNECTION TO () ?

proc sql;
connect to impala (&impala.);
execute by impala (set PARQUET_FALLBACK_SCHEMA_RESOLUTION=name) ;
create table SASDATASET as select * from connection to impala
  (SELECT * FROM db_name.tablename )
;
quit;

Did you trying issuing multiple statements at once with the EXECUTE() statement?

proc sql;
connect to impala (&impala.);
execute by impala (
set PARQUET_FALLBACK_SCHEMA_RESOLUTION=name;
CREATE VIEW IF NOT EXISTS &BASETABLE. as
(SELECT * FROM db_name.tablename )
) ;
create SASDATASET as select * from connection to impala
(select * from &BASETABLE. )
;
quit;

 

View solution in original post

2 REPLIES 2
Tom
Super User Tom
Super User

Are you saying the Impala cannot figure out how to names the variables it reads from a parquet file?

 

Can you try setting that option as part of the connection string? Will it stay in effect for your whole session once set?

 

Did you try issuing the SET statement with EXECUTE() before issuing the query with CONNECTION TO () ?

proc sql;
connect to impala (&impala.);
execute by impala (set PARQUET_FALLBACK_SCHEMA_RESOLUTION=name) ;
create table SASDATASET as select * from connection to impala
  (SELECT * FROM db_name.tablename )
;
quit;

Did you trying issuing multiple statements at once with the EXECUTE() statement?

proc sql;
connect to impala (&impala.);
execute by impala (
set PARQUET_FALLBACK_SCHEMA_RESOLUTION=name;
CREATE VIEW IF NOT EXISTS &BASETABLE. as
(SELECT * FROM db_name.tablename )
) ;
create SASDATASET as select * from connection to impala
(select * from &BASETABLE. )
;
quit;

 

PRAVIN_JAIN
Calcite | Level 5
As suggested below query worked, mistake I was making was to define option while creating the view instead of while fetching the data.

proc sql;
connect to impala (&impala.);
execute by impala (set PARQUET_FALLBACK_SCHEMA_RESOLUTION=name) ;
create table SASDATASET as select * from connection to impala
(SELECT * FROM db_name.tablename )
;
quit;

SAS INNOVATE 2024

Innovate_SAS_Blue.png

Registration is open! SAS is returning to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team. Register for just $495 by 12/31/2023.

If you are interested in speaking, there is still time to submit a session idea. More details are posted on the website. 

Register now!

How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Get the $99 certification deal.jpg

 

 

Back in the Classroom!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 2 replies
  • 363 views
  • 0 likes
  • 2 in conversation