Tag: Delta Lake

  • Direct Lake in Microsoft Fabric — The Problem, The Solution & How It Works

    Introduction — The BI Bottleneck That Held Enterprises Back

    For more than a decade, business intelligence systems have been built around a simple but painful truth: Power BI dashboards need refresh cycles. Whether scheduled hourly, daily, or weekly, refreshes were a necessary evil. They consumed compute, duplicated data, introduced latency, and forced teams to maintain complex incremental refresh logic.

    This model worked—until data volumes exploded, real-time expectations increased, and organizations demanded analytics that were instant, accurate, and always fresh.

    Microsoft Fabric introduces the breakthrough that finally ends this cycle: Direct Lake.

    Direct Lake is not an incremental improvement. It is a fundamental shift in how BI systems operate. It eliminates refreshes entirely, removes data duplication, and allows Power BI semantic models to read Delta tables directly from OneLake with warehouse-grade performance.

    SECTION 1 — The Problem Direct Lake Was Built to Solve

    Before Direct Lake, Power BI had two primary storage modes:

    1. Import Mode

    • Data is copied into Power BI
    • Dashboards rely on cached data
    • Refresh cycles required
    • Incremental refresh logic needed
    • High storage consumption
    • High compute cost for refreshes

    2. DirectQuery Mode

    • No data copies
    • Queries executed live against the source
    • Slow for large datasets
    • High load on the source system
    • Limited DAX capabilities
    • Limited modeling flexibility

    Both modes had strengths—but both had serious limitations.

    SECTION 2 — What Is Direct Lake?

    Direct Lake is a new storage mode in Power BI that allows semantic models to read Delta Lake tables directly from OneLake without importing or querying data through an engine.

    It combines the best of Import and DirectQuery:

    Comparison table:

    FeatureImportDirectQueryDirect Lake
    Data copiesYesNoNo
    Refresh cyclesYesNoNo
    PerformanceFastSlowFast
    Real-timeNoYesYes
    Source loadNoneHighNone
    Storage costHighLowLow

    Direct Lake is the first mode that delivers: No refreshes, No duplication, No source load, High performance, Real-time analytics.

    SECTION 3 — How Direct Lake Works (The Technical Deep Dive)

    To understand Direct Lake, you must understand three core components:

    • 1. OneLake — Fabric’s unified data lake. All workloads operate directly on OneLake.
    • 2. Delta Lake — The open storage format used for structured data. Supports ACID transactions, schema evolution, and high-performance reads.
    • 3. Semantic Models — Power BI’s modeling layer — measures, relationships, hierarchies, RLS/OLS.

    Direct Lake connects these three layers directly.

    The Magic: No Engine Between Power BI and Delta

    In Import mode, Power BI loads data into its internal VertiPaq engine. In DirectQuery mode, Power BI queries the source engine (SQL, Spark, etc.).

    In Direct Lake mode: Power BI reads Delta files directly from OneLake using a high-performance columnar reader optimized for analytics.

    No engine. No duplication. No refresh. No latency. This is why Direct Lake is so fast.

    Active design system:

    Colors

    Colors: black #000000, cyan-bluish-gray #abb8c3, white #ffffff, pale-pink #f78da7, vivid-red #cf2e2e, luminous-vivid-orange #ff6900, luminous-vivid-amber #fcb900, light-green-cyan #7bdcb5, vivid-green-cyan #00d084, pale-cyan-blue #8ed1fc, vivid-cyan-blue #0693e3, vivid-purple #9b51e0, base #FFFFFF, contrast #111111, accent-1 #FFEE58, accent-2 #F6CFF4, accent-3 #503AA8, accent-4 #686868, accent-5 #FBFAF3, accent-6 color-mix(in srgb, currentColor 20%, transparent)

    Fonts

    Fonts: Albert Sans, Alegreya, Arvo, Bodoni Moda, Bricolage Grotesque, Cabin, Chivo, Commissioner, Cormorant, Courier Prime, Crimson Pro, DM Mono, DM Sans, DM Serif Display, Domine, EB Garamond, Epilogue, Fahkwang, Figtree, Fira Sans, Fjalla One, Fraunces, Gabarito, IBM Plex Mono, IBM Plex Sans, Ibarra Real Nova, Instrument Serif, Inter, Josefin Sans, Jost, Libre Baskerville, Libre Franklin, Literata, Lora, Merriweather, Montserrat, Newsreader, Noto Sans Mono, Nunito, Open Sans, Overpass, PT Serif, Petrona, Piazzolla, Playfair Display, Plus Jakarta Sans, Poppins, Raleway, Roboto, Roboto Slab, Rubik, Rufina, Sora, Source Sans 3, Source Serif 4, Space Mono, Syne, Texturina, Urbanist, Work Sans, Manrope, Fira Code

    Element styles

    Element styles: button (color: #FFFFFF, bg: #111111, font: inherit, weight: inherit); h6 (weight: 700); heading (weight: 400); link (color: currentColor)

    📚 Direct Lake Series

    → Next: Direct Lake — Why It Matters & How It Compares to Import and DirectQuery

    🔗 Go Deeper

    Direct Lake Deep Dive Series

    This is Part 1 of 5 in the Direct Lake Deep Dive Series: Direct Lake in Microsoft Fabric — The Problem, The Solution & How It Works.

    Next → Direct Lake — Why It Matters & How It Compares to Import and DirectQuery

    Related Reading

  • OneLake — What It Is, Core Principles & How It Compares to Traditional Data Lakes

    Introduction — The Data Fragmentation Problem Enterprises Couldn’t Escape

    For years, organizations have struggled with fragmented data ecosystems. Data lived everywhere — Azure Data Lake Storage, Amazon S3, Snowflake, on-prem SQL servers, Hadoop clusters, and dozens of BI extracts scattered across teams. Every system had its own storage, its own governance model, its own security rules, and its own refresh cycles.

    This fragmentation created massive challenges, including:

    • Data duplication
    • Inconsistent security
    • Slow analytics
    • High operational overhead
    • Complex ingestion pipelines
    • Multiple versions of truth
    • Siloed teams
    • Expensive refresh cycles

    Microsoft Fabric introduces the solution that finally breaks this cycle: OneLake — the single, unified, organization-wide data lake.

    OneLake is not just storage. It is the foundation of the entire Fabric platform. It is the backbone that unifies data engineering, data science, warehousing, BI, and real-time analytics under one architecture.

    Section 1 — What Is OneLake?

    OneLake is Microsoft Fabric’s single, unified data lake for the entire organization. It is built on top of Azure Data Lake Storage (ADLS) but extended with Fabric-specific capabilities that make it more powerful, more integrated, and more open.

    What OneLake Is

    OneLake is:

    • Organization-wide
    • Delta Lake-native
    • Open format
    • Fully governed
    • Fully integrated
    • Zero-copy
    • Multi-engine
    • Multi-workload

    What OneLake Is Not

    OneLake is not:

    • A separate storage account
    • A BI cache
    • A warehouse
    • A Spark cluster
    • A dataflow container

    It is the single source of truth for all analytics workloads.

    Section 2 — The Core Principles Behind OneLake

    OneLake is built on four foundational principles:

    1. One Lake for the Entire Organization — Every workspace, Lakehouse, Warehouse, dataset, and pipeline stores data in OneLake.
    2. Open Delta Lake Format — All structured data is stored as Delta tables — open, ACID-compliant, and optimized for analytics.
    3. Zero-Copy Architecture — Power BI, SQL, Spark, ML, and real-time workloads all read the same Delta tables.
    4. Unified Governance — Purview handles lineage, sensitivity labels, access control, and classification across all workloads.

    These principles eliminate fragmentation and unify the entire analytics estate.

    Section 3 — OneLake vs Traditional Data Lakes

    Traditional data lakes (ADLS, S3, GCS) are powerful — but they are isolated. They require:

    • Separate compute engines
    • Separate governance tools
    • Separate security models
    • Separate ingestion pipelines
    • Separate BI refresh cycles
    • Separate ML environments

    How OneLake Solves These Challenges

    OneLake solves these problems by being:

    • Integrated — Every Fabric workload operates directly on OneLake.
    • Open — Delta Lake format ensures compatibility with Spark, SQL, ML, and BI.
    • Unified — One security model. One governance layer. One storage system.
    • Zero-Copy — No duplication across systems.
    • Multi-Engine — Spark, SQL, Power BI, ML — all read the same data.

    This is why OneLake is not "just another data lake." It is the analytics backbone.

    OneLake Series Navigation

    Previous in this series

    No previous post (this is Part 1 of 5).

    Next in this series

    OneLake Architecture Deep Dive — How It Works & The Power of Shortcuts

    Related posts

    Series Navigation

    Series: OneLake Deep Dive Series (Part 1 of 5)

    Next in this series

    OneLake Architecture Deep Dive — How It Works & The Power of Shortcuts

    Related Reading

  • OneLake Architecture Deep Dive — How It Works & The Power of Shortcuts

    Section 4 — OneLake Architecture — How It Actually Works

    OneLake is built on ADLS Gen2, but Fabric adds several layers on top:

    1. Delta Lake Storage Layer

    All structured data is stored as Delta tables: ACID transactions, Schema evolution, Time travel, Partitioning, Z-Order, File compaction.

    2. Fabric Namespace Layer

    Every workspace becomes a folder in OneLake.

    Example path: /OneLake/WorkspaceName/LakehouseName/Tables/Gold/Sales

    3. Multi-Engine Access Layer

    Spark, SQL, Power BI, ML, and Event Streams all read the same Delta files.

    4. Governance Layer

    Purview applies: Sensitivity labels, Lineage, Access control, Classification.

    5. Shortcut Layer

    OneLake can reference external data without copying it.

    6. Security Layer

    RBAC applies consistently across all workloads.

    This architecture is what makes OneLake unified, governed, and scalable.

    Section 5 — Shortcuts — The Most Underrated Feature of OneLake

    Shortcuts allow OneLake to reference external data sources without copying the data.

    Supported shortcut sources: ADLS, Amazon S3, Other OneLake workspaces.

    Why Shortcuts Matter

    • Zero duplication
    • Zero ingestion cost
    • Zero synchronization
    • Zero refresh cycles
    • Unified governance
    • Unified security
    • Unified lineage

    Shortcuts turn OneLake into a virtualized global data lake.

    Example Use Cases

    • Reference S3 data from a Lakehouse
    • Reference ADLS data from a Warehouse
    • Reference another team’s Lakehouse without copying
    • Build BI models on external data without ingestion

    Shortcuts are the key to eliminating data silos.

    OneLake Series — Part 2 of 5

    Previous in series
    OneLake — What It Is, Core Principles & How It Compares to Traditional Data Lakes

    Next in series
    OneLake + Delta Lake — The Perfect Combination Powering Every Fabric Workload

    Related posts

    📚 OneLake Series

    ← Previous: OneLake — What It Is, Core Principles & How It Compares to Traditional Data Lakes | → Next: OneLake + Delta Lake — The Perfect Combination Powering Every Fabric Workload.

    🔗 Go Deeper

  • OneLake + Delta Lake — The Perfect Combination Powering Every Fabric Workload

    Why Delta Lake is perfect for OneLake

    • ACID transactions
    • Schema evolution
    • Time travel
    • Partitioning
    • High-performance reads
    • Optimized for columnar analytics
    • Compatible with Spark, SQL, ML, and BI

    Delta Lake is the foundation that makes OneLake fast, reliable, and open.

    SECTION 7 — How OneLake Powers Every Fabric Workload

    OneLake is not a separate service. It is the storage layer for all Fabric workloads.

    How OneLake powers every Fabric workload

    1. Lakehouses — Store files + Delta tables directly in OneLake.
    2. Warehouses — Store SQL tables as Delta in OneLake.
    3. Power BI — Reads Delta tables directly via Direct Lake.
    4. Pipelines — Ingest data into OneLake.
    5. Notebooks — Transform data stored in OneLake.
    6. Event Streams — Write streaming data into OneLake.
    7. ML Models — Train directly on Delta tables in OneLake.
    8. Governance — Purview governs OneLake centrally.

    This is why OneLake is the backbone of Fabric.

    OneLake Series Navigation

    Previous in this series

    OneLake Architecture Deep Dive — How It Works & The Power of Shortcuts

    Next in this series

    OneLake for BI, Engineering, Warehousing, Real-Time & Governance — The Complete Picture

    Related posts

    📚 OneLake Series

    ← Previous: OneLake Architecture Deep Dive — How It Works & The Power of Shortcuts | → Next: OneLake for BI, Engineering, Warehousing, Real-Time & Governance — The Complete Picture.

    🔗 Go Deeper

  • Medallion Architecture & Delta Lake in Microsoft Fabric Lakehouse

    Lakehouse Deep Dive, Part 2: Medallion Design, Storage, and Delta Lake

    This post is Part 2 of the Lakehouse series and focuses on three areas: Medallion Architecture as the Lakehouse’s natural design pattern, Lakehouse storage using files and Delta tables, and Delta Lake as the engine that powers the Lakehouse.

    Medallion Architecture: The Lakehouse’s Natural Design Pattern

    This section covers Medallion Architecture and explains why it is a natural design pattern for the Lakehouse.

    Bronze, Silver, and Gold Explained

    This part of the deep dive focuses on Bronze, Silver, and Gold layers and how they are explained within the Medallion Architecture.

    Why Medallion Works Perfectly in Fabric

    This part describes why the Medallion Architecture works particularly well in Fabric.

    Lakehouse Storage: Files and Delta Tables

    This section of the deep dive covers Lakehouse storage, focusing on files and Delta tables.

    Two Types of Assets

    This part describes two types of assets in Lakehouse storage: files and Delta tables.

    When to Use Each

    This part explains when to use files and when to use Delta tables in the context of Lakehouse storage.

    Delta Lake: The Engine Behind the Lakehouse

    This section focuses on Delta Lake as the engine that powers the Lakehouse.

    ACID Transactions

    This part covers ACID transactions in Delta Lake.

    Schema Evolution

    This part covers schema evolution in Delta Lake.

    Time Travel

    This part covers time travel capabilities in Delta Lake.

    Partitioning

    This part covers partitioning in Delta Lake.

    Z-Order

    This part covers Z-order in Delta Lake.

    File Compaction

    This part covers file compaction in Delta Lake.

    Open Format

    This part covers Delta Lake as an open format.

    In the next post in this Lakehouse series, we will move from concepts to implementation and focus on SQL Endpoint, Direct Lake, and building a Lakehouse step by step.

    Lakehouse Series Navigation (Part 2 of 5)

    Previous in Series

    What Is a Lakehouse? — And Why It Replaced Data Lakes & Warehouses

    Next in Series

    SQL Endpoint, Direct Lake & Building a Lakehouse Step-by-Step in Microsoft Fabric

    Related Posts

  • What Is a Lakehouse? — And Why It Replaced Data Lakes & Warehouses

    Lakehouse Foundations · Part 1 of 5

    Why the Lakehouse Became the New Standard

    Modern analytics teams have been stuck between two imperfect options for years: fast but rigid data warehouses, or flexible but chaotic data lakes. Both solved important problems, but neither could deliver a complete, governed, end-to-end analytics platform on its own. This post kicks off a five-part series on the Lakehouse — and how Microsoft Fabric turns the idea of a unified analytics platform into a practical reality.

    The Old World: Lakes vs. Warehouses

    For a long time, organizations had to choose between two very different analytics worlds — and live with the trade-offs.

    Data Lakes

    Data lakes are flexible, scalable, and cost-effective. They are ideal for raw data, machine learning workloads, and handling unstructured assets.

    But when it comes to SQL analytics, strong governance, and reliable BI performance, traditional lakes fall short. The result is often a messy, hard-to-govern environment.

    Data Warehouses

    Data warehouses are structured, governed, and fast. They excel at dimensional modeling and powering business intelligence reports.

    However, they struggle with unstructured data, streaming scenarios, and very large-scale transformations, making them a poor fit for many modern analytics needs.

    This split forced teams into an uncomfortable reality: two storage systems, two compute engines, two governance models, two security layers, two ingestion pipelines, and ultimately, two competing versions of the truth.

    What Is a Lakehouse?

    The Lakehouse emerged to resolve this tension by combining the best properties of both lakes and warehouses in a single architecture.

    A Lakehouse brings together the flexibility of a data lake, the structure of a warehouse, the performance of columnar storage, the openness of Delta Lake, and the governance capabilities of enterprise systems.

    Just as important is what a Lakehouse is not. It is not a warehouse simply sitting on top of a lake. It is not a Spark cluster with SQL bolted on. It is not a BI model that happens to use lake storage. And it is certainly not just a marketing term.

    Instead, a true Lakehouse is a unified architecture where raw files, Delta tables, SQL endpoints, Spark notebooks, BI models, and machine learning workloads all operate on the same data, in the same place, under the same governance model.

    Fabric’s Lakehouse is one of the cleanest implementations of this concept in the industry, bringing these capabilities together in a way that is both powerful and approachable for modern analytics teams.

    The Fabric Lakehouse: What Makes It Different

    Microsoft Fabric takes the Lakehouse idea further than any platform before it. At its core is OneLake, a unified storage layer that serves the entire organization. On top of OneLake, the Fabric Lakehouse brings together storage, compute, governance, and BI in a single experience.

    Key components of the Fabric Lakehouse include:

    • Files — Support for raw data, logs, JSON, CSV, Parquet, images, PDFs, and more.
    • Delta tables — Structured, ACID-compliant, optimized tables that sit directly in OneLake.
    • SQL endpoint — A fully managed SQL engine that queries Delta tables without copying data.
    • Notebooks — PySpark, SQL, Markdown, and ML libraries, all working against the same underlying data.
    • Direct Lake integration — Power BI reads Delta tables directly, eliminating refresh cycles and data duplication.
    • Unified governance — Microsoft Purview provides lineage, labels, and access control in one place.
    • Unified security — Role-based access control is applied consistently across Spark, SQL, BI, and ML workloads.
    • Unified workspaces — Clear Dev/Test/Prod separation, deployment pipelines, and artifact organization.

    The result is not a “Spark-first” or “SQL-first” system. Fabric’s Lakehouse is a unified analytics engine where different personas can work the way they prefer, without fragmenting data or governance.

    Why Lakehouses Matter: The Business Perspective

    Technical details only matter if they move the business forward. Executives and business leaders care about outcomes: faster insights, lower costs, better governance, real-time analytics, reduced complexity, and a unified data strategy. The Lakehouse directly supports all of these goals.

    • Lower costs — Eliminates duplication across separate lake, warehouse, and BI systems.
    • Faster development — One platform, one storage layer, and one security model shorten the path from idea to insight.
    • Real-time analytics — Direct Lake integration removes traditional refresh cycles and copies.
    • Stronger governance — Purview governs data, analytics, and BI assets centrally.
    • Enterprise scalability — Delta Lake supports massive datasets without sacrificing performance or reliability.
    • Unified collaboration — Data engineers, BI developers, and data scientists all work on the same data, instead of maintaining separate pipelines.

    In other words, the Lakehouse is not just a technical evolution. It represents a fundamental shift in how organizations think about data, analytics, and governance across the business.

    What Comes Next in This Series

    This first post set the stage by explaining why the Lakehouse has become the new standard for modern analytics, and how Microsoft Fabric delivers a particularly strong implementation.

    In the rest of this five-part series, we will build on this foundation and explore how the Lakehouse model shapes architecture, development workflows, and analytics experiences across the organization.

    By the end of the series, you will have a clear mental model for how a Lakehouse works in practice and how Fabric can support your data, BI, and AI strategy on a single, unified platform.

    Lakehouse Series Navigation

    Previous in this series

    No previous post.

    Next in this series

    Medallion Architecture & Delta Lake in Microsoft Fabric Lakehouse

    Related posts

My journey from Datum to Data