Keeping pace with the ever changing data and AI landscape is a significant challenge, as new technologies, methodologies, and trends emerge rapidly. Keeping up to speed online is often like trying to hear yourself think at a football match. Distracting and often disappointing... depending on who you support! Industry events offer a unique alternative in that you can browse at your own pace, discuss your challenges and ambitions with select product experts, and even try your hand at the very best of emerging technologies with very little time or effort taken out of your day. But with so many events to choose from, it’s hard to pinpoint which would be the most valuable for you to attend.   Introducing Big Data London, the UK’s biggest free to attend data, analytics and AI event. It’s attended by almost 20,000 people each year and features a diverse range of expert speakers from leading organizations, offering insights into cutting-edge developments across data management, analytics, AI, and more. Additionally, Big Data London provides numerous opportunities for hands-on learning, networking, and collaboration, making it a must-attend event for professionals seeking to stay ahead in the fast-evolving data and analytics landscape. SAS have once again chosen to sponsor Big Data London to showcase the latest developments in our software and provide you with the opportunity to experience one of our many product demos. From our new product launch Viya Workbench, to exploring how to enable trust and transparency in your models, to transforming analytics through high-quality synthetic data generation, there’s much to see and much to learn.   So, come and join us at Big Data London this year. Registration is free and you can find us at stand Y554... follow the data flow to the pink Viya cube. Our friendly team are on hand for an informal conversation or even a deep dive into your data needs. Or visit either one of our speaking sessions:   Harnessing the Power of Generative AI: Practical Applications and Insights Transformative Power of Data and Analytics in Government, SAS & DWP   See you there! ... View more

Decision makers in cloud-savvy organisations continuously seek ways to reduce unpredictability of cost for consuming cloud services, while aiming to maximise their return on investment. Those investing in solutions based on Lakehouse architecture, such as Databricks, anticipate cost savings and productivity gains from this unified storage solution for their cloud-migrated data. After all, Delta Lake storage, which underpins the Lakehouse architecture, combines the strengths of both Data Lakes and Data Warehouses, leveraging the inherent flexibility of the cloud.   However, what if certain business operations require processing the data in Lakehouse, using an application that is not co-located with it? Does this mean the data must be transferred to the remote application? This scenario would introduce concerns about network latency, performance, and egress cost. Alternatively, should organisations consider replacing these remote applications with new, co-located ones? This approach also seems impractical, as it would be very disruptive and counterproductive to the anticipated business objectives.   Some recently published blog posts by my colleagues on SAS Communities (see reference links below) describe how SAS can harness the analytical power of data stored in Databricks. For example, Cecily Hoffritz discussed in her blog how user-friendly interface of SAS Viya enables more users within an organisation to use the Databricks Lakehouse and participate in data analysis, decision-making, and innovation, regardless of their technical background. In another example, Patric Hamilton explained in his blog how SAS Data Quality can be applied to the Databricks Lakehouse for Entity Resolution, enhancing the effectiveness and accuracy of data-driven decisions. I will continue with the theme in this blog and describe how can organisations extract even more value from their investment in Databricks through the “In-Database processing” capabilities of SAS.   As the name suggests, “SAS In-Database“ processing allows processing to happen inside the database to utilise resources much more efficiently and effectively. Examples of SAS In-Database processing with Databricks include Implicit and Explicit SQL Pass-Through, In-Database Procedures  and In-Database Model Scoring facilities. In-Database Scoring for Databricks enables the environment to leverage Massive Parallel Processing architecture of the database and allows processing to happen locally at the database level. Only the final result travels to network, that too if required. The function remains in the SAS language and is executed by a lightweight SAS engine (SAS Embedded Process for Spark) deployed within the Databricks cluster. Deployment of SAS Embedded process is a one-time task for the administrator, and a simplified view of the overall process could look as shown below.     Once SAS Embedded Process has been deployed, users can develop models using SAS Studio (by writing SAS code), SAS Model Studio or SAS Intelligent Decisioning (by creating visual pipelines). SAS In-Database scoring supports the following platforms to publish and run the models in Databricks:   Databricks on AWS or on Azure Spark Table (as of version 2023.10) Amazon S3 Microsoft ADLS Gen2     Here is a list of steps and example code to publish a model to a Spark table and run it in Azure Databricks:   Step 1: Start a CAS session and assign a caslib to Spark cas mysess; caslib myspark datasource=( srcType="spark", platform="databricks", server="myserver.azuredatabricks.net", userName="token", password="authentication-token", clusterId="cluster-id", jobManagementUrl="https://nnnnn.cloud.databricks.net/", httpPath="my-http-path", schema="my-schema" );   Step 2: Publish a model to a Spark table proc scoreaccel sessref=mysess; publishmodel exttype=databricks caslib="myspark" modelname="mymodel" storefiles="/myfiles/mystore.ast" programfile="/myfiles/myprogram.sas" modeldatabase="mydatabase" ; run; quit;   Step 3: Start a Databricks interactive session before you run the model proc cas; sparkEmbeddedProcess.startSparkEP caslib="myspark"; run; quit;   Step 4: Run the model proc scoreaccel sessref=mysess; runmodel exttype=spark caslib="myspark" modelname="mymodel" modeldatabase="mydatabase" intable="mytable" outtable="mytable_out"; run; quit;   Step 5: Stop the SAS Embedded Process for Spark continuous session proc cas; sparkEmbeddedProcess.stopSparkEP caslib="myspark"; run; quit;   Conclusion SAS In-Database processing enables organisations to extend the value of their data stored in a Databricks cluster by executing programs so that data is analysed without crossing the database boundary – giving them the benefits of improved performance, productivity and governance, while avoiding the data egress cost.   Learn more about SAS and Databricks Harness the analytical power of your Databricks platform with SAS Data everywhere and anyhow! Gain insights from across the clouds with SAS Elevated efficiency and reduced cost: SAS in the era of Cloud Adoption SAS and Databricks: Your Practical Guide to Data Access and Analysis Data to Databricks? No need to recode - get your existing SAS jobs to SAS Viya in the cloud Maximize Coding and Data Freedom with SAS, Python and Databricks Data Brilliance Unleashed: SAS Data Quality against Databricks - Precision, Performance, Perfection Unlock Seamless Efficiency: SAS Viya's No-Code/Low-Code Experience to Democratize Databricks ... View more

Adoption of cloud computing in recent years has dramatically changed the way organisations handle data and analytics these days. Explosive growth of generative AI based applications and services is expected to drive the use of public cloud infrastructure even further. With such a decisive shift towards a more dynamic and scalable IT environment, organisations who designed their critical business applications based on traditional SAS deployment (aka SAS9 customers) are looking for ways to leverage exciting opportunities offered by the cloud while causing minimum disruption to their well-established workloads.   SAS too has swiftly evolved in recent years to align itself with this exciting new paradigm with the introduction of SAS Viya. This is an open and highly performant AI, Analytics & Data Management Platform that runs as a containerised application in a Kubernetes cluster in both public and private cloud environments. At the same time, the industry has witnessed rise of new data lake platforms from vendors like Databricks, Snowflake, Starburst and more.   With a steady growth in expenditure on cloud infrastructure and services every year, senior executives are increasingly looking for ways to get more for their money as they modernise their analytics workloads. Not surprisingly, managing cloud expenses has emerged as a more pressing challenge than security for the second year in a row,  according to a recent State of the Cloud report.    Figure 1: Survey results for challenges faced in migrating workloads to public cloud (Source: Flexera 2024 State of the cloud report)   These cost considerations are not just limited to storage and compute costs but also include evaluation of multi-cloud strategy and other factors such as cost of migration (re-coding and re-testing established complex applications in a new language) cost of training business experts to become expert programmers (a blended approach of low-code/no-code and pro-code) efficiency gains from a unified platform that brings different together different personas to work with consistent data   Cost savings with improved performance When it comes to the infrastructure cost, findings in this performance benchmark report  published by an independent consultancy concluded that SAS Viya is one of the most cost-performant and cost-conscious analytics platform available to enterprise cloud users. The cloud-native architecture of SAS Viya means that cost savings can also be achieved with features like in-database processing and workload management. This gives the ability to minimise data movement and optimise the cloud resources to remain active or scale up during high workload periods and to shut down or scale down during periods of reduced activity.   Mitigating cost of migration Migrating a well-established application developed in SAS9 may have been an intimidating thought but SAS has a proven track record of exceptional customer support and this proud legacy continues to be at the core of our discussions, when we plan a move to SAS Viya with our SAS9 customers. You can get a good sense of the complexity and effort involved for a planned migration by using SAS Content Assessment tool. Moreover, this case study brings a customer experience of migrating from SAS9 to life and this blog demonstrates how you can reuse majority of your SAS9 code in SAS Viya, based on our analysis of over 3 Billion lines of code (as of May 2023).   Efficiency gains While cost represents one critical consideration in the migration planning, it is worth keeping a close eye on the efficiency gains that the modern platform has to offer. In this context, the ability of SAS Viya to support programming experts (SAS as well as open source) and non-programmers from the same platform means that different personas can collaborate effectively. The “beautifully designed interfaces across the entire data-to-decision lifecycle” (quoted from the Forrester’s AI Decisioning report) of SAS Viya enable business users and non-programmers to generate valuable insights from data with a self-service approach. A blended use of SAS Studio Steps, SAS code, Python code and Custom Steps in the same Studio Flow, as shown below, is a great example of how significant efficiency gains can be realised by bringing users with diverse skills together.     Figure 2: A view of SAS Studio user interface in SAS Viya (Source: SAS)   Conclusion The transition to this new world, with cloud taking the centre stage, presents both opportunities and challenges for organisations and it is critical that they must evaluate the technology options carefully. In this context, it is crucial that the SAS9 customers make sure their view is not blurred by well-established concepts presented with new terminology or by vendors comparing the latest, designed-for-cloud technology with the traditional SAS9 capabilities.  Organisations will only fully understand their options, costs and potential benefits when comparing new vendors’ offerings with SAS Viya Technology.   Other blogs in the series: Harness the analytical power of your Databricks platform with SAS - SAS Support Communities Data everywhere and anyhow! Gain insights from across the clouds with SAS   ... View more

SAS Information Catalog – Bring governance for your data and AI assets together   Data is undoubtedly one of the most important assets for an organisation, and effective governance of this valuable asset helps organisations meet their regulatory obligations as well as innovate faster. Those who have implemented governance best practices for their data assets can trust their data and explain its journey through the entire decisioning process. Organisations are increasingly adopting Data Catalogs to support their governance-led initiatives and improve transparency in their business. A recent research report by Fortune Business Insights, projects the global data catalog market size to grow from $878 Million in 2023 to £3.4 Billion by 2030, at a CAGR of 21.5% during the forecast period.   Governance for data and AI While governance of data assets has remained high on the agenda for several years, extending the governance remit to AI assets is now seen as equally important by data driven organisations. With openness, trust and governance as its core values, SAS Viya is a unique decisioning platform that brings together all aspects of analytics lifecycle – making it easy for different user personas to work with data at different stages and collaborate effectively. This ability to orchestrate the entire analytics lifecycle from one unified platform has enabled SAS to exclusively bring governance for all information assets under one roof  through the SAS Information Catalog.     While data engineers, data scientists and business analysts can use SAS Information Catalog to search for relevant data assets for their analysis and reporting requirements, they can also use the catalog to find dataflows (data pipelines), models (both SAS and open source), rules, decisions and more. Business users can search for relevant reports while data stewards and IT administrators can detect duplicate assets for improved overall governance.   Federated governance for all data sources (#DataMeshUsers are you listening?) SAS Viya provides transparent read/write access to data from any source or platform. This includes leading cloud data platforms, such as Azure Synapse, AWS Redshift, Google BigQuery, Snowflake, Databricks, and relational and nonrelational databases (Oracle, Teradata, MongoDB) – as well as support for open big data file formats (Apache ORC, Apache Parquet) etc. Regardless of the source or where data resides, Data Discovery Agents in SAS Information Catalog enable users to search through the entire data landscape from one unified catalog and find the relevant resource quickly. This can be particularly useful when implementing a Data Mesh methodology, where data product owners can rely on the Information Catalog as the one-stop shop for users from all business domains.   Finding the right resource for your need (Know your data and use it with confidence) Powerful search capabilities of SAS Information Catalog enable users to narrow down their search for information assets to the most relevant list, so that they can quickly access and evaluate asset fitness, avoid rework, and make informed choices. The search results can be based on any combination of keywords, metadata facets or extended attributes such as status, asset owner or tags. Search results provide detailed information on each data asset – including data health, data quality, key metrics, etc., at both table and column levels. Data exploration views, which include correlation and outlier detection information, help users understand the data profile and column analysis to quickly determine if the data asset is fit for purpose before they begin any detailed analysis on it. Data Discovery Agents’ ability to analyse column semantic type is particularly useful for data audit and regulatory compliance as data stewards can use this facility to detect the existence of any personal data in a data asset. Lineage View is another key example of governance with SAS Information Catalog, which shows a business-level impact analysis of information assets (including open source models) in the context of their sources, their outputs, and the relationship between them.     Working with Glossary Starting from February 2024, a Glossary component has been added to the SAS Information Catalog,  which enables users to manage terms and associate them with information assets registered in the catalog. This provides a consistent source for term definitions that can be used by business and technical users alike. SAS customers who have used SAS Business Data Network in earlier software versions will quickly identify the potential this offers for improved overall governance.   Extending SAS Information Catalog with REST API SAS Information Catalog REST API was published in the October 2023 release, and serves as the programming interface to the Catalog Service. The API calls can be triggered from any programming language and offer a great way to programmatically find and select what you need fast from vast amounts of data. This enables SAS metadata to be shared with any enterprise catalog and can also be used to enforce an organisation’s governance policies by triggering jobs when certain conditions are met. One of my personal favourite examples of using this REST API involves adding new asset types or new attributes to existing assets (e.g. data from usage logs) to meet bespoke business requirements. This opens a fascinating new set of possibilities in which SAS Information Catalog can make data driven organisations even more productive while maintaining high standards for information governance. ... View more