Introducing Databricks LakeFlow: A Unified Solution for Data Engineering

Databricks, the Data and AI company, has launched Databricks LakeFlow, a comprehensive solution designed to simplify and unify all aspects of data engineering, from ingestion to transformation and orchestration. LakeFlow enables data teams to efficiently ingest data at scale from databases like MySQL, Postgres, and Oracle, as well as enterprise applications such as Salesforce, Dynamics, SharePoint, Workday, NetSuite, and Google Analytics. Additionally, the new Real Time Mode for Apache Spark™ supports ultra-low latency stream processing.

LakeFlow automates the deployment, operation, and monitoring of data pipelines at scale in production, featuring built-in CI/CD support and advanced workflows that include triggering, branching, and conditional execution. Integrated data quality checks and health monitoring systems, such as PagerDuty, ensure reliability. This solution simplifies the creation and operation of production-grade data pipelines, even for the most complex data engineering tasks, meeting the increasing demand for reliable data and AI.

Addressing Data Engineering Challenges

Data engineering is crucial for democratizing data and AI within businesses but remains a challenging field. Data teams often deal with siloed, proprietary systems, requiring complex connectors, and intricate data preparation logic. Operational disruptions due to failures and latency spikes can lead to customer dissatisfaction. Deploying and monitoring pipelines involves using disparate tools, complicating the process. Current solutions are often fragmented and incomplete, resulting in low data quality, reliability issues, high costs, and a backlog of work.

LakeFlow simplifies these challenges by providing a unified experience built on the Databricks Data Intelligence Platform, with deep integrations with Unity Catalog for governance and serverless compute for efficient and scalable execution.

Key Features of LakeFlow

– LakeFlow Connect: Enables scalable data ingestion from various sources using native connectors for databases like MySQL, Postgres, SQL Server, and Oracle, as well as enterprise applications like Salesforce, Dynamics, SharePoint, Workday, and NetSuite. These connectors are integrated with Unity Catalog for robust data governance and utilize the efficient capabilities of Arcion, acquired by Databricks in November 2023, to support batch and real-time analysis.

– LakeFlow Pipelines: Automates real-time data pipelines using Databricks’ Delta Live Tables technology, allowing data transformation and ETL in SQL or Python. It supports low-latency streaming without code changes, unifies batch and stream processing, and offers incremental data processing for optimal performance. This feature simplifies complex data transformations, making them easy to build and operate.

– LakeFlow Jobs: Automates workflow orchestration across the Data Intelligence Platform, handling scheduling of notebooks and SQL queries, ML training, and automatic dashboard updates. It provides enhanced control flow capabilities and full observability to detect, diagnose, and mitigate data issues, increasing pipeline reliability. LakeFlow Jobs centralizes the deployment, orchestration, and monitoring of data pipelines, helping data teams meet their data delivery commitments.