Watch this Ask the Expert session to learn the importance of data catalog and other data governance areas to improve delivery of analytic insights.    Watch the webinar   You will learn: Some of the common challenges hindering or decreasing the analytic pipeline delivery and business benefits. How to get reliable, trusted data quicker to maximize benefits from the data for reporting or analysis. How to integrate open source products.   The questions from the Q&A segment held at the end of the webinar are listed below and the slides from the webinar are attached.   Q&A  Are there any options to integrate or share metadata with other applications?  I did not go there, but thanks for asking the question. One thing I should mention is yes, we are working on that, and we currently have a way to do that as well. We have integrated with this open-source metadata standard called Egeria. Egeria is part of the Linux Foundation. It's an open metadata standard, more like a metadata marketplace, where different application vendors can share their metadata. It benefits the organization, so you do not have to go and stitch all these together, right? Right now, we are contributing our metadata from the SAS information governance product to Egeria. If the data stored in the organization has enabled, it automatically events all the assets that the user sees in the product to Egeria, which is open metadata standard and can be combined with other application metadata so that way they can see this lineage across different vendors. Obviously, we are also looking at integrating with open lineage and markets, which is other open metadata standard tools, to augment the lineage capability. We wanted to address the analytic use case with the discovery and lineage and be able to partner and integrate with other tools as well, whether it is open source and creating this innovation and build a partnership. The simple answer is yes, we can. This material is available to extract to the third-party and we are also able to take the third-party metadata into SAS using rest APIs and building that Azure pipeline. We build on top of that by integrating with the other vendors.    What role does the cataloged data play in Lineage?  The group, but I should also mention that we do not want to see lineage as a separate piece. That is one of the key things we are carrying out as part of this discovery. The admin is not just building the discovery Cadillac for users to identify datasets, they are also in the background. We are building the lineage meaning. As the data has been cataloged, we also capture the dataset usage. Whether the dataset used in the report or the model, everything is captured. The relationship between one dataset to another asset is being captured as part of that cataloging as well. What that means is when a user discovers the asset here, they could also go to lineage and discover the asset as well. Same exact asset, but more towards the left or right discovery of lineage.    Where can I find this Catalog Home? I cannot see it under SAS Drive. Is it something that our organization needs to activate?  If you look at the side menu bar, discover information assets. That is the SAS information governance and catalog the product. This is the discovery of information assets.    Can SAS Information Catalog discover CAS Lib and Compute Lib?  If you are existing SAS users, you are aware of the SAS library and CAS library where you could look at SAS datasets. It does both. In fact, all the data set catalog that SAS table and CAS table. This is, you know, a traditional SAS table with SAS7BDAT format. I can look at that individually or CAS table, which is the SASHDAT for the in-memory engine that can be cataloged as well. But both can be available together for the users.    Does this platform run on prem and or in the cloud?  This is native cloud deployment whether private or public cloud. If there is an on Prem cloud private cloud, you could deploy this as well. Can we deployed on multiple public clouds, although we have partnered with Azure for better integration. This is a cloud deployment, and it is multi cloud supported.    Is information governance part of SAS Viya or should it be licensed additionally?  That is important as you start looking at using this. The information catalog product packaging wise is any visual package or bundles that, as part of SAS Viya, will get the basic version of the catalog product. Basic version would have the search and be able to explore and see some of the profile metrics. That is any visual SAS Viya products along with the SAS Visual Analytics. SAS models would contain the basic version of catalog because regardless of whether they buy the reporting product or analytics product, they would want to discover it. We want to give the ability of the basics as information. But they can add additional capabilities, such as the semantic Identification or Information privacy. There are others, such as machine learning, business summary identification, and language detection and workload. Those are part of the sales information governance license. The basic is included with all the bundles, which is the sense information catalog. The license is the SAS information governance with advanced features.    Can cloud data be accessed?  Yeah, of course. Given this is a cloud-native deployment application, regardless of what cloud it is. You can easily call any cloud data on Azure, whether SQL Azure Synapse or Azure blob store. Same goes for AWS - AWS S3, Redshift or Yanmar Yanmar. Amazon's Hadoop version symbols of GCP for Google, as well. Any cloud data can be crawled and cataloged and be made available for discovery.    What features are available to help with the consumption of these data assets? For example, is there a way to combine columns from different sources into a logical view for consumption by applications?  The simple answer is yes and that's part of the action. When we were exploring the data assets in the action part, we have multiple actions. Prepare data is the action use can take while using SAS Information Governance to do data preparation. One data could be coming from Snowflake in the cloud and other data could be coming from a data warehouse sitting in the on premise. Regardless of where it is and if the connector is established, all kinds of transformation could be applied. Column can be removed, or column can be transformed. All of those preparations can be done at the prepared data layer and you can create this analytic based table, or one table more for preparation by the data engineer (or in some cases the analyst).    Does information governance allow for metadata updates by various stakeholders in a decentralized manner or is it more of a centralized Data management method?  If integration with other metadata sources, 3 rd party vendors and/or sharing metadata was meant by decentralized then yes, it is. SAS Viya can share metadata with other players using integration with Egeria open metadata standard.    Recommended Resources SAS Information Governance page Getting started with SAS Information Catalog in SAS Viya SAS Information Governance Release Highlights Please see additional resources in the attached slide deck.   Want more tips? Be sure to subscribe to the Ask the Expert board to receive follow up Q&A, slides and recordings from other SAS Ask the Expert webinars.   ... View more

Presenters Kumar Thangamuthu, SAS; Nancy Rausch, SAS Abstract SAS continues to enhance its data management functionality to provide a comprehensive and integrated set of capabilities for collecting, transforming and managing your data. New features include enhanced capabilities for working with data from a wide variety of environments and types, including Hadoop, cloud environments, RDBMS, files, unstructured data, streaming and others. There are new automation and discovery enhancements, as well as new support for building flows and cataloging assets to understand the context and governance of useful data in an enterprise. Watch the presentation Watch What’s New in SAS® Data Management on the SAS Users YouTube channel.   ... View more

The popularity of cloud data storage has grown exponentially over the last decade as more and more organizations are transitioning from on-premises to cloud data storage and data management. Microsoft Azure is one of the big players accelerating the move to cloud. In this article, we will look at the overview of SAS® to access data in Azure Storage and Big Data.   As organizations start to realize the impact of digital transformation, they are moving storage to the cloud as they move their computing to the cloud. Data Storage in the cloud is elastic and responds to demand while only paying for what you use, similar to compute in the cloud. Organizations must consider data storage options, efficient cloud platform and services, and migrating SAS applications to the cloud.   Organizations can connect from the SAS platform and access data from the various Azure data storage offerings. Whether it is Azure Data Lake Storage or Big Data HDInsight, SAS/ACCESS engines and data connectors have these covered with optimized data handling abilities to empower organizations going through digital transformation journey.   SAS AND AZURE DATA LAKE STORAGE SAS Viya can read and write ORC and CSV data files to Azure Data Lake Storage Generation 2(ADLS2). There is a new data connector called SAS ORC Data Connector to facilitate the data transfer between CAS and ADLS2. The SAS ORC Data Connector enables you to load data from an Apache Optimized Row Columnar table into CAS. This data connector can be used with a path or Azure ADLS2 CASLIB.   SAS LIBRARY TO ADLS /*create a CAS session */ cas casauto; caslib "Azure Data Lake Storage Gen 2" datasource=( srctype="adls" accountname=”sasdemo” filesystem="data" dnsSuffix=dfs.core.windows.net timeout=50000 tenantid=<Azure Application Tenant ID UUID> applicationId=<Azure registered application UUID> ) path="/" subdirs global /* creates library reference for SAS compute */ libref=AzureDL; caslib _all_ assign; Here is an explanation of the parameters that are used to create a caslib: CASLIB – A library reference. The caslib is the space holder for the specified data access. The Azure Data Lake Storage Gen 2 CAS library is used to specify the ADLS data source. SRCTYPE – Source type is ADLS which corresponds to Azure Data Lake Storage connection. ACCOUNTNAME – Azure Data Lake Storage account name. FILESYSTEM – ADLS container file system name. TENANTID & APPLICATIONID – Available from the Azure Registered Application page for your organization or individual use. PATH – Points to the directory structure where the file system resides. LIBREF – Creates a SAS library reference along with a CAS library. SAS Viya uses the available CAS session CASAUTO to create the CAS library reference to ADLS. In this example, it uses a CAS clustered environment with 3 nodes, including 2 worker nodes. To make the CAS library available to all the users, global parameters can be used. ORC or CSV can be loaded from the ADLS blob container file system “data” to a CAS in-memory cluster or saved to ADLS from CAS.   LOAD AND SAVE ORC DATA TO AZURE STORAGE FROM SAS VIYA Let’s look at an example to load a SAS dataset to a CAS in-memory server. Once the data is in CAS, it can be used for any distributed data processing, report, analytics, or modeling. The final CAS in-memory data output is saved as an ORC file to Azure Data Lake Storage. proc casutil; load data=sashelp.class casout="class" outcaslib="adls" replace; save casdata="class" casout="class.orc" replace; quit; Important parameters: PROC CASUTIL – CASUTIL procedure that works with CAS tables including data transfer, table & file information, and drop & delete tables. LOAD – CAS action set to load data to CAS in-memory servers DATA – SAS traditional data set used as input CASOUT – Output CAS in-memory distributed table OUTCASLIB – Output CAS Library reference SAVE – CAS action set to save CAS in-memory table to ADLS CASDATA – Input CAS table CASOUT – ORC output file saved to ADLS   SAS AND AZURE HDINSIGHT HADOOP DATA LAKE           Traditionally, the Hadoop platform has been deployed to solve an organization’s Data Lake business needs on-premise. With more and more incoming data along with deploying analytical applications such as SAS Viya in the cloud, organizations are moving some of their Data Lake needs to the cloud along with the data. Azure HDInsight is one such Hadoop service that provides a cloud native Hadoop platform along with elasticity to scale up or down the Data Lake. The SAS platform can leverage Azure HDInsight Hadoop service to access, load, or save data using the SAS Access to Hadoop engine.   LIBNAME STATEMENT TO CONNECT AND ACCESS DATA option set = SAS_HADOOP_CONFIG_PATH = "/opt/hdinsight/conf"; option set = SAS_HADOOP_JAR_PATH = "/opt/hdinsight/jars"; options sastrace = ',,,sd' sastraceloc = saslog no$stsuffix fullstimer; libname hdilib HADOOP uri='jdbc:hive2://sgfdemo.azurehdinsight.net:443/default;ssl=true?hive.server 2.transport.mode=http;hive.server2.thrift.http.path=hive2' server='sgfdemo.azurehdinsight.net' user="admin" pw="demopassword" schema="default"; As a one-time configuration setup, before creating a SAS library reference and connecting to an Azure HDInsight cluster, the Hadoop administrator would execute a HadoopTracer Python script available from SAS to extract all the necessary JAR and config files. The SAS administrator can further use these extracted files to connect to an Azure HDInsight cluster by setting two option variables. This step is performed regardless of the type of Hadoop distribution or cloud platform.   Some of the important parameters: SAS_HADOOP_CONFIG_PATH - Specifies the directory path for the Azure HDInsight cluster configuration files. SAS_HADOOP_JAR_PATH - Specifies the directory path for the Azure HDInsight JAR files. URI – Azure HDInsight JDBC URI to connect to Hive server 2. Once a similar JDBC URI is retrieved from Azure HDInsight documentation, just modify the HDInsight server name. The JDBC URI contains some of the necessary parameters that are enabled and assigned values by default. SSL is set to true by default, and REST transport mode is set to HTTP. Data can be loaded from an Azure HDInsight cluster to the SAS platform or saved to the cloud. SAS PROCs such as Proc Append, Sort, Summary, Means, Rank, Freq, and Transpose are supported on Azure HDInsight cluster. Furthermore, the DATA step and PROC SQL data preparation with bulk load are handled to save data efficient on the cloud.   SAS can access data directly from Azure Storage and Big Data or through other Azure intermediary data sources that uses Azure ADLS2 or HDInsight as the persistent storage layer. As the end users migrate from on-premise to cloud, they also asses the data storage landscape instead of just lifting and shifting to similar data source in the cloud to address Self-Service data management. Variety of data types such as structured, unstructured and the need to access data faster are some of the key deciding factors to choose cloud blob or big data storage options. ... View more

Many common data platforms already in use are being refactored and delivered as service offerings to Azure cloud customers. Azure offers database service technologies that are familiar to many organizations. It is important to understand the terminology and the different database services to best meet the demands of your business use case or application. Benefits to organizations are reducing hardware and software footprint to manage. Databases that scale automatically to meet business demand and software that optimizes and creates backups means organizations can spend more time deriving insights from their data and less time managing infrastructure.   SAS provides efficient SAS Data Connectors and SAS In-Database Technologies support to Azure database variants. A data storage running in Azure cloud is much like an on-premise database, but instead Microsoft manages the software and hardware. Azure services can take care of the scalability and high availability of the database with Database as a Service (DBaaS) offerings and minimal user input. SAS integrates with Azure cloud databases whether SAS is running on-premise or in the cloud.   SAS AND AZURE SQL DATABASE          Organizations can connect and access data from an Azure SQL Database using SAS/Access to Microsoft SQL Server from SAS. All the features from SAS/Access to SQL Server running on-premise would be available in the cloud as well. Running the SQL database in the cloud gives organizations elasticity to scale the database. Let’s look at code samples to connect and access data from using SAS library and CAS library.   LIBNAME STATEMENT TO CONNECT TO AZURE SQL DB           Azure SQL Database connection information typically specified in the odbc.ini file on the SAS servers, along with a data source name (DSN). Specify the DSN name in the libname connection parameter.   libname mydblib sqlsvr noprompt="uid=sgfdemo; pwd=demopassword; dsn=Azure SQL Database;" stringdates=yes; The SQLSVR connection parameter enables the LIBNAME statement to use SAS/Access to SQL Server to connect to Azure SQL Database.   SAS VIYA CAS LIBRARY TO CONNECT TO AZURE SQL DB          Similarly, SAS/Access to SQL Server can be used to connect to an Azure SQL Database using a data connector. This approach is helpful to organizations with streamlined data movement when SAS Viya is running in the cloud to load and save data between Azure SQL Database and SAS Viya in-memory servers.   cas casauto; caslib azsqldb desc='Microsoft SQL Server Caslib' dataSource=(srctype='sqlserver' username='sas' password='demopassword' sqlserver_dsn="Azure SQL Database" catalog='*'); Azure SQL Database connection information has to be added to the odbc.ini files in all the CAS cluster nodes. This facilitates optimized data movement between multiple CAS worker nodes and Azure SQL Database.   SRCTYPE – Source type SQLSERVER is the key parameter to connect to an Azure SQL Database using SAS Viya Data Connector to Microsoft SQL Server.   Load and Save SAS Dataset to Azure SQL DB           With one table in the Azure SQL Database, let’s load another table from a SAS Viya in- memory cluster to the Azure SQL Database. In the following code sample, first we are loading a SAS data set named cars to a CAS in-memory cluster. Then the distributed CAS in-memory table is saved to the Azure SQL Database.   proc casutil; load data=sashelp.cars outcaslib="azsqldb" casout="cars" replace; save casdata="cars" casout="cars" replace; quit; One parameter to mention here, OUTCASLIB, contains the name of the CAS library that we created earlier to access the Azure SQL Database. Information about other parameters has been furnished in the ADLS section above.   SAS VIYA AND AZURE MYSQL DATABASE           MySQL is another widely used relational database management system that is accessible from SAS Viya as an Azure MySQL database with elastic server configurations to support various data volumes. SAS Data Connector to MySQL can be used to access, load, and save data between CAS in-memory cluster nodes and an Azure MySQL Database.   Other flavors of MySQL databases such as Azure MariaDB Database can also be accessed using SAS Data Connector to MySQL. Steps are exactly the same as MySQL, just modify the server host to Azure MariaDB, along with the appropriate credentials.   cas casauto; caslib azurems desc='MySQL Caslib' dataSource=(srctype='mysql' host='sgfdemo.mysql.database.azure.com' username='sas@sgfdemo' password='demopassword' database="demo") /* global */ ; The MySQL client must be installed on all the CAS cluster nodes. The MySQL client library path must be added to the CAS configuration settings file on the nodes.    SRCTYPE – Source type MYSQL is the key parameter to connect to an Azure MySQL Database using SAS/Access to MySQL.   LOAD AND SAVE SAS DATASET TO AZURE MYSQL DATABASE proc casutil; load data=sashelp.cars outcaslib="azurems" casout="cars" replace; save casdata="cars" casout="cars" replace; list files; quit; SAS Viya uses SAS Data Connector to MySQL to save a CAS in-memory table named cars to an Azure MySQL Database. The Save CAS action, which is part of the TABLES CAS action set has been called to perform a serial save in this case.   SAS VIYA AND AZURE POSTGRESQL DATABASE         The SAS platform can access data from another popular relational database management system(RDBMS), PostgreSQL. Azure PostgreSQL Database is a Database as a Service (DBaaS) offering to support data movement as needed through various server configurations based on data volume and demand.   SAS LIBRARY TO LOAD SAS DATASET libname azurepg postgres server="sgfdemo.postgres.database.azure.com" port=5432 user=sas@sgfdemo password='demopassword' database=postgres; /*Load a SAS dataset to Azure PostgreSQL database*/ data azurepg.cars; set sashelp.cars; run; LIBNAME statement uses the POSTGRES keyword to utilize SAS/Access Engine to PostgreSQL to connect to an Azure PostgreSQL Database. In this example, a SAS dataset named cars is loaded to the destination cloud PostgreSQL database.   CAS LIBRARY TO AZURE POSTGRESQL cas casauto; caslib postgrescaslib desc='PostgreSQL Caslib' dataSource=(srctype='postgres' server='sgfdemo.postgres.database.azure.com' username='sas@sgfdemo' password='demopassword' database="postgres"); The CAS library postgrescaslib is created by connecting to the destination Azure PostgreSQL Database. Source type POSTGRES is the keyword that utilizes SAS Data Connector to PostgreSQL.   Load and Save SAS Dataset to Azure PostgreSQL DB proc casutil; load data=sashelp.class outcaslib="postgrescaslib" casout="class" replace; save casdata="class" casout="class" replace; list files; quit; SAS Viya uses SAS Data Connector to PostgreSQL to save the CAS in-memory table named cars to the Azure PostgreSQL Database. The Save CAS action, which is part of the TABLES CAS action set has been called to perform a serial save in this case. The LIST FILES action in the CASUTIL procedure show the cars table has been saved to the database in the cloud.   Once the data is loaded to CAS in-memory servers for distributed parallel processing or SAS compute server, advanced analytics and model methods can be applied to get relevant insight out of the prepared data. Whether the final decision is to keep data on-premise or in the cloud or a hybrid approach, SAS Data Connector supports virtually any data sources to access, load, move, model data efficiently, or navigate through the analytic life cycle. It is important to think about the use case for your database and the type of data you plan to store before you select an Azure database service. ... View more

  Modern Data Lake The data lake idea focuses on storing all analyzable data sets in raw or lightly processed form into the easily expandable scale-out Hadoop infrastructure to ensure that the fidelity of the data is preserved. The modern data lake will be a “managed” data lake, meaning one that uses a data lake management platform to manage data ingestion, apply metadata and enable data governance so that users know what’s in the lake and can use the data with confidence. These factors help to avoid data lake turning in to a “data swamp”.   Modern managed data lakes allow businesses to explore their data quickly and easily, identifying opportunities for business and process improvements across the organization as well as to better understand their customers. Cybersecurity and personal data protection play vital role in every organization. This is a very big data problem as threats come from internal and external sources of an organization. The General Data Protection Regulation (GDPR) is intended to standardize expectations and protect personally identifiable information on employees, clients, and applicable data subjects. This means cybersecurity data collection and analysis has to become proactive and always on. Data governance is important in securing the data in data lake with organization’s procedures and policies that manage the data usage, availability, quality, privacy and security of data.   Data Lake Market Growth   Streaming data from high-velocity, high-volume sources from small to large companies contributing to explosive growth of data lake market. Streaming data can be   Constantly generated, in small bursts like IoT, product logs Accumulated quickly – Billions of records occupying hundreds of petabytes Unstructured or semi-structured in its original form   Data Lake growth drivers   Evolution of data lakes on cloud such as Amazon S3 or Azure data lake store that can offload the maintenance effort, take care of auto-scaling, encryption and ease of management Availability of choice and better accessibility with the emergence of data lake querying tools based on SQL and ad-hoc querying Varied pricing options for agile businesses, especially with the separation of storage and compute methodologies in data lake. Desire to give data scientists access to enterprise data for exploration, discovery, and insight creation   Effect of latest trends on Data Lake   Trends like IoT, AI, live streaming data have created enormous amount of data being thrown into data lakes. Though managing a data lake or data hub to derive business insights used to be expensive, moving data lakes to the cloud have eased some of those pains, along with elastic infrastructure. Cloud-based data lakes tend to offer better availability than what is guaranteed on-premises, faster time to value and time to deploy for new projects, data sources and applications already cloud-based and scalable. Data Lakes are evolving to handle different modes of data movement (streaming for example) and different formats of data that are either consumed or generated by the IoT or AI projects.   Data Lake Vs Data Warehouse   Data moves through a data lake much faster than a data warehouse, reducing latency and providing faster time to analyzed data. Data lakes uses “schema-on-read” approach, accelerating insights and lowering the cost of acquiring new types of high-volume data, sensor data, social media data and clickstream data. Data in data lakes are continuous and not confined to periodic updates, as with data warehouses, allowing for real-time analysis from streaming data. Though there are significant differences between data lakes and data warehouses, they both complement each other in an enterprise environment. A data lake becomes a landing and staging area for the data warehouse and other downstream applications. Meanwhile, data warehouse serves as a repository of integrated, historical data that is dimensionally modelled and can be queries in an ad hoc manner by concurrent business users. Data Lakes provide flexibility in the ability to apply structure to data after it has arrived in the Data Lake. Data warehouses often demand well-defined structures up front into which data is fit.   Limitations of Data Lake   Data lakes lack strong governance, metadata, security and other standard features of enterprise software, though managed data lakes are starting to address some of these limitations. As companies move beyond piloting advanced analytics projects to run data lake in production and at scale, they have found the software tools quite expensive, they also require investments in people and products required to manage the open source functionality and infrastructure.   Data Lake play in industry verticals   The basic nature of the data lake concept is industry agnostic. A data lake is a storage system that can store large amounts of data in its original format until required by advanced analytic and visualization applications to derive insights. Verticals simply differ by how they use the data, in other words, how the data is transformed to be used by a specific organization or industry.   For example, government initiatives like Smart Cities, that can manage traffic, optimize power grids, enhance education systems, track vehicle patterns, connected vehicles and much more generate massive amounts of data. Data lakes and analytics can assist cities in building more sustainable, livable environments.   Banking industry is also moving fast towards making the most of growing and monetizing data. Entering the big data universe, banks have now come to rely on data lakes to bring all the data together and accelerate the data-to-action cycle. Banks accelerate the pace to go from data to insightful reports using Hadoop-based data lake synched with visualization and analytics tools.     SAS & Data Lake           SAS is a leader in big data management and advanced analytics solutions that help customers make better decisions, faster. The following capabilities, all supported by SAS, can enable data management and analytics for data lake and other valuable repositories:   Data integration - Data movement, in-database processing, and native data access to traditional and emerging data sources such as Hadoop.   Data quality - Cleanse, standardize, and enrich data in real time and batch with prebuilt rules.   Self-service big data preparation - Business users profile, cleanse, and transform data on Hadoop without writing code.   Business glossary and metadata management - Track lineage, business rules, descriptive details, and workflow for improved governance of your data assets.   Event stream processing - Analyze real-time streams of data in motion for better decisions.   Data virtualization - Provide blended, secure views of your data without moving it.   Hadoop support - Access, deliver, and process data inside Hadoop across both the data management and analytics life cycle.   Visualization and advanced analytics - Deliver cutting-edge visualization and analysis capabilities without requiring analytical skills. ... View more

  When we talk about Data Lake, Hadoop is synonymous with the medium of implementation. Data in Hadoop can be accessed from SAS using SAS/ACCESS to Hadoop and SAS/ACCESS to ODBC. SAS/ACCESS to Hadoop and SAS/ACCESS to ODBC, each have their own place in a data lake, but using the former has inherent advantages over the later, ODBC. With the introduction of SAS/ACCESS to Spark (comes with SAS/ACCESS to Hadoop), it is even more appealing to use SAS/ACCESS to Hadoop. Here are 10 benefits of using SAS/ACCESS to Hadoop vs SAS/ACCESS to ODBC.   SAS/ACCESS Interface to Hadoop reads data directly from the Hadoop Distributed File System (HDFS) when possible to improve performance. This differs from the traditional SAS/ACCESS engine behavior (ODBC), which exclusively uses database SQL to read and write data.    There are more CREATE TABLE options available for ACCESS to Hadoop. A paper on it:            http://support.sas.com/resources/papers/proceedings17/SAS0409-2017.pdf              ACCESS to ODBC only supports: DBCREATE_TABLE_OPTS=            ACCESS to Hadoop has many options that assist in creating tables, such as   DBIDIRECTEXEC ·  DB CRE A TE _TA B LE _OP TS = · POST_STMT_OPTS= · POST_TABLE_OPTS= ·  PRE_STMT_OPTS= ·  PRE_TABLE_OPTS=                  The most important “secret” that you need to know about the SAS create table options is that they enable you to specify              text strings that are placed into CREATE TABLE and CREATE TABLE AS SELECT (CTAS) statements that SAS generates.              Each option is responsible for placing the text string in a different location in the SQL statement. POST_STMT_OPTS= is              interesting because its behavior changes base on whether DBIDIRECTEXEC has been enabled.   This covers ACCESS to Vertica vs ACCESS to ODBC but there are some good points in there (ignore INSERT/UPDATE/DELETE):              https://communities.sas.com/t5/SAS-Data-Management/SAS-Access-to-Vertica/td-p/130915   Connections are easier, because they are DSN-less (SERVER=, PORT=, etc.)   Advantages of using SAS/ACCESS to Hadoop, with respect to data transfer Streaming read/write is supported. This means moving data is much faster than ODBC.   More HiveQL functions are eligible for passing to Hive.   Better internationalization support with ACCESS to Hadoop.   In the SAS Viya space you get Multi-node support with the CAS Data Connector   Better integration with Hadoop High Availability technology. For example, the libname to connect to a Hive HA enabled system can be as simple as “libname x hadoop”.   PROC SQOOP support - (https://blogs.sas.com/content/sgf/2015/10/28/using-sas-proc-sqoop/)                     PROC SQOOP procedure enables users to access an Apache Sqoop utility from a SAS session to transfer data                       between a database and HDFS. Using SAS PROC SQOOP lets you submit Sqoop commands from within your SAS                   application to your Hadoop cluster.                     PROC SQOOP is licensed with SAS/ACCESS® Interface for Hadoop, it’s not part of the Base SAS® license. PROC                   SQOOP is supported in UNIX and Windows SAS.   Bulk loading is better using SAS/ACCESS to Hadoop Vs ODBC, one of the main reason is because, when BULKLOAD=YES parameter is set, the engine uses HDFS to load data into Hive   Advantages of SAS/ACCESS to Hadoop with respect to In-Database processing, SAS In-database Procedure support (SUMMARY/MEANS, REPORT, SORT, TABULATE, FREQ). Notice, RANK is not supported.     Most importantly ACCESS to Hadoop supports In-Database Technologies for Hadoop, really vital to avoid data movement to do Data Management activities. SAS Embedded Process in Hadoop plays a key role here, which ACCESS to Hadoop makes use of. Some areas of importance here are, (***Following are not supported with SAS/ACCESS to ODBC***)   If you are planning to run SAS EP based process at Hadoop Cluster, SAS/ACCESS interface to Hadoop is must. The SAS EP based process let you use the Hadoop cluster resources to run data calculation at source and improve the performance. It also let you run selective SAS PROCS at Hadoop cluster.    If you are planning to run SAS Data cleansing and Standardization operation at Hadoop cluster using QKB, these are SAS EP based process and you have to have the SAS/ACCESS interface to Hadoop to execute it.   If you are planning to use SAS Data-loader for Hadoop (DL4H) software against a Hadoop cluster, you have to have SAS/ACCESS interface for Hadoop and SAS EP to support the DLH operations.   If you are planning to read parallel data from Hadoop cluster to SAS LASR/CAS, you have to have the SAS/ACCESS interface for Hadoop with SAS EP.   Hadoop optimized SQL pushdown using PROC SQL   Support for PROC HDMD to create the metadata file in Hive library, stored in HDFS. This is useful for faster direct data load from HDFS and running In-Database scoring.   Code Accelerator support for Hadoop. Makes use of Hive and HDMD. Again, this is really important when data movement is of concern, SAS code (DS2) can be pushed to Hadoop for processing.   Beginning in 9.4M6 and Viya 3.4, In-Database Technologies for Hadoop includes In-Database Technologies for Spark. SAS Data Connect Accelerator for Spark SAS Scoring Accelerator for Spark SAS Embedded Process for Spark            Requirements for SAS In-Database Technologies for Spark          SAS In-Database Technologies for Spark requires SAS In-Database Technologies for Hadoop, which is separately licensed.           The SAS Embedded Process for Spark is included with SAS In-Database Technologies for Spark. The SAS Embedded             Process must also be installed on the Spark cluster to enable SAS Data Connect Accelerator for Spark. SAS recommends         installing the latest version of the SAS Embedded Process.           For information about supported versions and requirements: Click here   The following applies to In-Database Technologies for Hadoop: For Viya 3.4: The platform option in the Hadoop caslib can be used to control whether the EP job is run using MapReduce (default) or Spark. This applies to both the Data Connect Accelerator and Scoring Accelerator. If platform=sparkis used, Scala code is generated by the application which is then submitted to the cluster to run as a Spark job.   Documentation: Click here   For SAS 9.4M6: The global option, HADOOPPLATFORM, is used to control how EP jobs are run. Setting this option equal to Spark causes the application to generate Scala code that describes the task to be run and submits that to the cluster as a Spark job. This isn’t described in external documentation yet but will be once M6 is available to customers.   In both cases, YARN Resource Manager UI can be used to examine how the job was run within the Spark execution environment if detailed information is needed. The SAS/CAS logs will show information about the results of the job.   In-Database Technologies for Spark is available in Viya 3.4 as a limited availability option.   Essentially SAS/ACCESS to ODBC connection has generic data access support. There are no Hadoop specific capabilities added or optimized. ... View more