Top Ten features of the Microsoft Intelligent Data Platform

To help your organization move forward quickly with its innovation strategy, we have put together the top ten innovation-focused features of the Microsoft Intelligent Data Platform to support a data-driven strategy.

Organizations all over the world are keen to strive for innovation with their data, and boardrooms all over the world are keen to become data-driven. However, when the executive leadership team issues directions to their technology teams, it can seem very complicated to get started. For example, in a world that is full of technology, it can be challenging to identify software features that will help to drive innovation or help derive value from data. What should a company look for in features? What’s important and what isn’t?

With the announcement of the Microsoft Intelligent Data Platform, we want to explore how the platform simplifies the data journey for businesses by aligning big data, governance and analytics in one place. This blog will also highlight the main features under these three pillars to help signpost people around the Microsoft Intelligent Data Platform.

Business users expect their reports to display data quickly to showcase their analytics. Therefore, a crucial aspect of the Microsoft Intelligent Data Platform is to ensure that the data is returned fast. SQL Server 2022 includes the next generation of Intelligent Query Processing (IQP)capabilities based on common customer challenges.

IQP is aimed at resolving query performance issues. It can help speed up large analytical type queries by giving you an approximation, versus having to scan large tables. In databases, scanning can be very time-consuming; it is the equivalent of reading through the entire phone directory to find a record versus a seek, where you use the index to find the records more quickly. To try it out, you can head over to Microsoft’s Github repository for sample code and data.

Data professionals often don’t like working with JSON data. Unfortunately, the challenge is that it is becoming increasingly difficult to avoid it! Both Azure SQL Database and SQL Server 2002 have improved capabilities to process JSON data. Microsoft now offers new and enhanced T-SQL function support for JSON capabilities that also returns data in JSON formatas well as importing data.

For developers and data professionals, it is easier to integrate data with other services to support REST endpoints. Using T-SQL, the developer can push data within the SQL Server engine or send it to ML services or any other REST endpoint. For IoT environments, Azure Event Hub could also be a target for the data since JSON is a common format for many devices. Many REST web services return results that are formatted as JSON text or accept data that are formatted as JSON, so this embeds SQL Server and Azure SQL Database technologies as part of wider architectures for AI, web and business applications as well as IoT architectures.

Often, developers need to create code both online and offline as well as work with data stores that could reside in the cloud or on-premises. For this reason, the Microsoft Intelligent data platform includes a public preview for a new local developer experience, helping developers to publish updates through CICD pipelines.

In addition, the new developer experience facilitates workstation or GitHub code spaces environments through Visual Studio Code extensions. It also supports the offline experience by providing an emulator for Azure SQL Database using containers. The new developer experience includes SQL bindings for Azure Function Integration and introduces new T-SQL JSON capabilities to meet the intricacies of data stores.

Eventually, we may hear less of ‘big data’ technologies and focus instead on ‘data’ technologies as the business is able to access data from the edge right to the cloud, thereby enabling insights in real-time as well as in a batch mode. SQL Server is now capable of querying SQL Data from the Edge to Cloud. This means that the SQL developer can use the same T-SQL language tools and core engine to access data.

Since SQL Server 2022 has machine learning capabilities as well as time-series analysis, there are great opportunities to make your data work for your organization regardless of whether it originates from the cloud or an on-premise server.

For example, the developer could take a backup of a database and restore it in a choice of locations, such as an IoT edge device or the cloud. The developer could even store it in containers. Ultimately, this enables the data to be stored in a variety of locations, powering opportunities for developers to become creative with their organizational data.

As part of a data-driven strategy, analytics is a crucial aspect of many aspects of the business, such as understanding customers better or cutting costs. The importance of analytics increases as organizations include IoT data sources, such as sensors or other devices.

Regardless of whether the data is IoT or not, organizations can query their data using the well-known and mature SQL language to produce extensive insights regardless of the data source.

From the business perspective, analytical insights are achieved more quickly so there is no need to learn a new language. This reduces friction and increases business adoption of IoT applications as well as data-based applications. For example, developers can deploy a SQL Server module to store data on a device that is running Azure IoT Edge with Linux containers. This means that developers can use Azure IoT Edge and SQL Server to store and query data at the edge using the T-SQL language, opening up opportunities for gathering and analysing more data from sensors.

For the purposes of data governance, Azure Purview offers insights and lineage of data, and it can also help to align policy management. This means it is a true hub for security as well as helping organizations to manage and understand their data.

Microsoft Purview is part of the public preview for Dynamic Lineage for SQL, helping with impact analysis to support security efforts. Purview includes useful assets for data governance such as data insights, data classification, and policy management that can extend across all the organization’s SQL assets and data estate. Azure Purview also offers data sharing facilities to organisations to help them securely share data with third-party organizations, opening up further opportunities to make it easier for businesses to capitalize on their data.

For businesses, it is crucial to understand their data sources well so that they can truly work towards deriving value from their data. Unfortunately, many businesses often do not have a grip on the data they have. In this case, how do they make the most of the opportunities within the data? Microsoft Purview tackles this issue by offering a central hub for data. One particularly interesting feature is the Dynamic Lineage feature which conducts ongoing analysis to track lineage dynamically. Currently, there is a preview release of dynamic lineage extraction from Azure SQL Databases in Azure Purview.

Businesses need analytics for their SQL operational data, but it can often involve creating complex Extract, Transform and Load (ETL) packages which are difficult to change easily. Following on from this innovation, the Microsoft Intelligent Data Platform meets the need for data to be ‘easy’ by introducing the Synapse Link for SQL.

Normally, ETL is typically the solution but it can be expensive. Worse, it may even adversely impact the primary workload on the operational database. Microsoft’s solution is Synapse Link for SQL, melding data between operational data and analytics data seamlessly. In the Microsoft Intelligent Data Platform, SQL Server 2022 and Azure SQL Database both support the Synapse link for SQL.

Developers also need to be able to track changes to data to meet business needs in a fluid, dynamic environment. To meet developer needs, Microsoft also introduced the general availability of Azure SQL Database Ledger. As developers become increasingly familiar with working with data, the Microsoft Intelligent Data Platform offers widespread availability of Azure SQL Database ledger to track and audit modification history or modifications to data.

They have automatic history and a ledger view to see all modifications and who made them. In addition, as regulations change and adapt worldwide, the ledger tables can be updated so they are ideal for protecting sensitive data for auditing and compliance purposes.

Blockchain technology is an exciting technology for data professionals and developers alike, but there are concerns over data governance and protection with this relatively new technology. Following on from the innovation in the Azure SQL Database ledger, there are enhancements in data protection. The Ledger provides ways that cybersecurity teams can cryptographically attest data to auditors and business parties so that any evidence of data tampering would be discovered.

This functionality provides proof that the data has not been tampered with by unauthorized access, thereby helping the organization to meet third-party regulatory audits and require legal compliance examinations for compliance certification such as SOX, PCCI, GDPR and so on.

Further, the Ledger comes with built-in crypto hashes of transactions for data protection in the form of a blockchain. In addition, a hash of the blockchain is stored separately to provide independent verification for multi-party trust verification of data. Cybersecurity experts can access the historical data for compliance, audit, forensics, or any other purpose as evidence to support independent verification.

The right software features are key to the success of your data-driven business strategy. With the right software, your innovation initiatives will be efficient and effective, disappearing into the background to enable the business to be successful.

To summarise, there is a range of great new features in the Microsoft Intelligent Data Platform, particularly in SQL Server 2022 that help the technology to be useful and supportive rather than becoming a side-show that ultimately distracts the business from achieving the company vision.

For Business Intelligence and Analytics experts, there are opportunities for real-time analytics without affecting production operational workloads with Synapse Link for SQL. One highlight is the new Azure-connected capabilities along with enhanced query intelligence and core engine features for security, scalability, and availability.

For developers using Azure SQL Database, there is a new developer experience and the ability to deliver faster with JSON data. We also looked at the Azure SQL Database Ledger, which is generally available and helps organizations make the most of blockchain technology.

Data is crucial to any business, and the Microsoft Intelligent Data Platform is making data easier for small, medium and large businesses alike. What innovation are you most excited about?

Powerful Business Intelligence solutions can transform your complex data into business insights to share across your organisation. Our data experts can help you plan, build and deliver your solutions to improve decision-making and business outcomes. Find out more about our Business Intelligence service or get in touch.