SAS IBM Biginsights integration - Best Practices

[tommy81]

Hi ,

We are looking for case studies on the best implementation pracitces to integrate sas and ibm biginsights 4.2.

Please share any papers , or lesson learnt if any.

Thanks

Tommy

[JBailey]

Hi @tommy81

I don't know of anything specifically for IBM BigInsights. There is however a lot of general information on using SAS with Hadoop.

This is the first page I would check - SAS 9.4 Support for Hadoop. The first link is a great place to start.

Can you tell me which SAS products you are planning to use with BigInsights? SAS 9.4 or SAS Viya? SAS/ACCESS Interface to Hadoop? Data Loader for Hadoop?

For now, I am going to assume that you are talking about SAS 9.4 SAS/ACCESS Interface to Hadoop. 

With SAS and Hadoop it is important to understand the systems requirements. You can find that information here:

SAS 9.4Supported Hadoop Distributions

Reading, and understanding, the SAS/ACCESS Interface to Hadoop documentation is a great place to start.

SAS/ACCESS 9.4 for Relational Databases: Reference, Ninth Edition

Pay special attention to the discussion on 32K strings and Hadoop. This is covered, briefly, in the SAS Data Types section. I am planning to post a SAS Communities article on this. It causes issues. If you ever see SASWORK run out of space, this could be the cause.

If you recieve an error message (specifically regarding security) when trying to connect to Hadoop using a LIBNAME statement, it may be a good ideas to try to run PROC HADOOP. It displays more detailed error messages regarding security.

Take a look at the SAS/ACCESS Documentation Page, pay attention to the Related Documentation and Technical Papers sections. There is a link to the SAS/ACCESS doc listed here. This is a great page to bookmark.

For information regarding SAS In-Database check out this page: SAS In-Database Technology.

Information on using SAS In-Database Procedures can be found in PART 6 of the SAS 9.4 In-Database Products: Users Guide. 

There is a common trap to avoid: it is widely assumed that SAS/ACCESS Interface to Hadoop can be used to access IBM BigSQL (which lives on Hadoop). It cannot. If your SAS environment needs to access IBM BigSQL then SAS/ACCESS Interface to ODBC should be used.  

I am in the process of putting publishing some workshop materials that I created a while back. I am planning to put the materials on Githup. I will try to remember to let you know when they are published. Feel free to ping me to ask how it is progressing.

[tommy81]

We would have SAS Data Loader for Hadoop . I believe SAS/ACCESS to Hadoop will be a part of this package in SAS 9.4

[JBailey]

Let me know if you need more informaion or have specific questions. If you need further help, holler.

[JBailey · Accepted solution]

The SAS Global Forum workshop materials from "An Insider's Guide to SAS/ACCESS to Hadoop" has been published on GitHub. 

https://github.com/Jeff-Bailey/SGF2016_SAS3880_Insiders_Guide_Hadoop_HOW

If you find this useful, let me know. I may add more content to it.

[tommy81]

For the SAS Stream to Hadoop , should we buy a separate license for the SPDE engine or would it be bundled along with the SAS DATA loader ?

Also is it the best method vis-a-vis the SAS/ACCESS to HADOOP  ??? ,since we need not go through hive to hit the hdfs file system.

[JBailey]

Hi @tommy81

The SPDE engine for Hadoop is included in Base SAS, so you should be good-to-go.

Whether it is best, or not, depends on what you want to do with it. 

Best wishes,
Jeff