Category: Microsoft Fabric

  • What Are Event Streams & Why They Matter in Microsoft Fabric

    ← Back to Microsoft Fabric — Complete Learning Series

    9.1 — What Are Event Streams?

    Event Streams is Fabric’s real‑time ingestion and transformation engine. It replaces an entire stack of traditional streaming infrastructure — Kafka, Spark Streaming, Flink, Kinesis — with a single, governed, low-complexity capability built into the platform.

    Key Capabilities:

    • Real‑time ingestion
    • Real‑time transformation
    • Real‑time routing
    • Real‑time monitoring
    • Real‑time dashboards
    • Direct integration with OneLake, Lakehouses, Warehouses, and Power BI

    Event Streams unify streaming and analytics under one platform. No separate infrastructure. No separate team to run it.

    Why Event Streams Matter — The Business Perspective

    The demand for real‑time insights has never been higher — retail wants live sales dashboards, logistics wants live shipment tracking, finance wants instant fraud detection, manufacturing wants sensor monitoring in real time. Traditional architectures made this painful. Event Streams make it simple.

    Business Benefits:

    • Real‑time decision making — instant insights, faster reactions, better outcomes
    • Lower costs — no separate streaming infrastructure to build or operate
    • Faster development — no Kafka, Spark Streaming, Flink, or Kinesis required
    • Unified governance — Purview governs streaming data centrally
    • Unified storage — streaming data lands directly in OneLake
    • Unified analytics — streaming + batch + BI in one platform

    Event Streams transform real‑time analytics from a complex engineering challenge into a simple, governed, scalable capability.

    ← Back to Microsoft Fabric — Complete Learning Series

  • Building Real-Time Dashboards & Governing Semantic Models in Microsoft Fabric

    ← Back to Microsoft Fabric — Complete Learning Series

    8.4 — Building Real-Time Dashboards

    Direct Lake makes real‑time BI simple. The architecture is straightforward:

    Event Streams → Lakehouse → Gold → Direct Lake → Dashboard

    Use Cases:

    • Real‑time KPIs and sales dashboards
    • Operational monitoring
    • IoT analytics
    • Fraud detection
    • Supply chain visibility

    No refresh jobs. No scheduled pipelines pushing data into Power BI. The Gold tables update, and the dashboard reflects it instantly.

    Semantic Model Governance — Purview Integration

    Semantic models aren’t just technical assets — they’re governed assets. Purview treats them as first-class citizens in the governance model.

    Purview Capabilities for Semantic Models:

    • Lineage — trace from source data through to the dashboard
    • Sensitivity labels — classify models as Confidential, Internal, Public, etc.
    • Access control — enforce who can use or edit a model
    • Classification — identify PII, financial data, and regulated content
    • Audit logs — track who accessed what and when
    • Policy enforcement — prevent unauthorized use automatically

    When a semantic model is governed, every report built on top of it inherits that governance. That’s what makes enterprise BI trustworthy.

    ← Back to Microsoft Fabric — Complete Learning Series

  • How Direct Lake Works — vs Import vs DirectQuery

    8.2 — How Direct Lake Works

    Direct Lake connects three layers seamlessly:

    1. OneLake

    Unified storage — one copy of data for the entire analytics estate.

    2. Delta Lake

    Open, ACID‑compliant tables stored as Parquet files with transaction logs.

    3. Semantic Models

    Power BI’s modeling layer reads directly from those Delta files — no engine in between, no copy, no cache refresh.

    That’s why it’s fast. The data never moves. The model just reads it.

    Direct Lake vs Import vs DirectQuery

    FeatureImportDirectQueryDirect Lake
    Refresh requiredYesNoNo
    Real‑timeNoYesYes
    PerformanceHighLowHigh
    Source loadNoneHighNone
    Data duplicationYesNoNo
    Storage costHighLowLow
    DAX supportFullLimitedFull
    Best forSmall datasetsOperational dashboardsEnterprise BI

    Direct Lake wins on every axis that matters for enterprise analytics. It’s not a compromise — it’s a genuine step forward.

  • Designing Semantic Models & Optimizing Delta Tables for Direct Lake

    8.3 — Designing Semantic Models for Direct Lake

    Direct Lake expects clean, structured, business-ready data. The model design is what separates fast dashboards from slow ones.

    Best Practices:

    • Use Gold tables — Direct Lake expects business‑ready data, never Bronze or Silver
    • Use Star Schema — fact + dimension tables, not wide flat files
    • Use Surrogate Keys — improve join performance
    • Use Aggregation Tables — pre‑compute KPIs to boost performance
    • Avoid Wide Tables — keep fact tables narrow
    • Use Hierarchies — improve drill‑down in reports
    • Use RLS / OLS — secure data access at the model level

    Semantic models must be clean and structured. The cleaner the Gold layer, the faster and more reliable the BI layer.

    Optimizing Delta Tables for Direct Lake

    Direct Lake performance is only as good as the Delta tables underneath it. Optimization is not optional at enterprise scale.

    Key Optimizations:

    • Partitioning — partition fact tables by date or region to limit scan size
    • Z‑Order — optimize columnar reads on frequently filtered columns
    • File Compaction — avoid tiny files that slow down reads
    • Vacuum — clean up old Delta files to reduce storage and improve performance
    • Schema Evolution — handle column changes safely without breaking downstream models
    • Merge Optimization — improve upsert performance for incremental loads

    A poorly optimized Gold table means slow dashboards regardless of compute capacity. Optimization is where real performance is won.

  • What Are Semantic Models & Direct Lake — The BI Revolution in Microsoft Fabric

    ← Back to Microsoft Fabric — Complete Learning Series

    8.1 — What Are Semantic Models?

    Semantic models are the brain of Power BI. They sit between raw data and dashboards — translating tables into business logic.

    They define:

    • Measures
    • Relationships
    • Hierarchies
    • Aggregations
    • RLS / OLS
    • Calculation groups
    • Metadata

    Without a semantic model, raw data is just tables. With one, it becomes a governed, reusable analytics layer that every dashboard in the organization can build on.

    Direct Lake — The Biggest Innovation in Power BI’s History

    For years, Power BI was stuck between two modes:

    • Import Mode — fast, reliable, but required scheduled refresh cycles
    • DirectQuery Mode — real‑time, but slow and limited

    Direct Lake eliminates the tradeoff entirely. Semantic models read Delta tables directly from OneLake — no refresh, no duplication, no latency.

    Benefits:

    • No refresh cycles
    • No data duplication
    • Real‑time dashboards
    • Lower storage cost
    • High performance (Import-class speed)
    • No incremental refresh logic
    • No scheduled jobs

    Direct Lake is the clear winner for enterprise BI. It combines the speed of Import with the freshness of DirectQuery — and adds the governance of OneLake on top.

    ← Back to Microsoft Fabric — Complete Learning Series

  • Medallion Architecture — Designing Bronze, Silver & Gold in Depth

    Designing the Bronze Layer — Raw Data

    Purpose

    Capture raw data exactly as it arrives.

    Characteristics

    • Minimal transformation
    • Raw schema
    • Raw files or raw Delta tables
    • High volume
    • Append‑only

    Best Practices

    • Store raw files in /Files/Bronze
    • Convert to Delta for consistency
    • Add ingestion metadata (timestamp, source)
    • Avoid business logic
    • Use Pipelines for ingestion
    • Use Event Streams for real‑time

    Common Bronze Sources

    • APIs
    • Databases
    • ERP systems
    • CRM systems
    • IoT devices
    • Logs
    • S3/ADLS shortcuts

    Bronze is your “source of truth.”

    Designing the Silver Layer — Cleaned & Conformed

    Purpose

    Transform raw data into clean, standardized, analytics‑ready tables.

    Best Practices

    • Use PySpark notebooks
    • Use Delta Lake merges
    • Apply schema enforcement
    • Remove duplicates
    • Standardize column names
    • Add surrogate keys
    • Partition large tables

    Common Silver Tasks

    • Remove nulls
    • Standardize date formats
    • Convert strings to numeric types
    • Join reference tables
    • Apply business logic
    • Flatten nested JSON

    Designing the Gold Layer — Business‑Ready

    Purpose

    Provide BI‑optimized, business‑friendly tables.

    Best Practices

    • Use SQL endpoint for modeling
    • Build fact + dimension tables
    • Use surrogate keys
    • Use incremental loads
    • Partition fact tables
    • Use aggregation tables

    Common Gold Tables

    • FactSales
    • DimCustomer
    • DimProduct
    • FactInventory
    • FactFinance
    • KPI tables

    Gold is your “analytics layer.”

  • Medallion Architecture in Microsoft Fabric — What It Is & Why It Matters

    ← Back to Microsoft Fabric — Complete Learning Series

    Introduction — Why Medallion Architecture Became the Standard

    Modern analytics systems must handle massive data volumes, real‑time ingestion, complex transformations, and business‑ready modeling — all while maintaining governance, performance, and reliability. Traditional ETL pipelines often collapse under this pressure, producing messy data, inconsistent schemas, duplicated logic, slow refresh cycles, and unreliable BI outputs.

    The Medallion Architecture solves this problem. Originally popularized by Databricks and now deeply embedded into Microsoft Fabric, the Medallion pattern organizes data into Bronze → Silver → Gold layers, each with a clear purpose, clear quality expectations, and clear transformation boundaries.

    1. What Is Medallion Architecture?

    Medallion Architecture is a layered data design pattern that organizes data into three quality tiers:

    Bronze — Raw Data

    • Ingested from source systems
    • Minimal transformation
    • Stored as files or raw Delta tables
    • Schema may be messy
    • Used for traceability and reprocessing

    Silver — Cleaned & Conformed

    • Standardized schemas
    • Deduplicated
    • Type‑casted
    • Enriched
    • Joined
    • Business logic applied

    Gold — Business‑Ready

    • Aggregations
    • Dimensional models
    • Star schemas
    • KPI tables
    • BI‑optimized structures

    2. Why Medallion Architecture Matters

    • Clarity — Each layer has a defined purpose.
    • Scalability — Large datasets become manageable.
    • Reliability — Silver and Gold layers are stable and predictable.
    • Governance — Purview can classify and label each layer differently.
    • Performance — Gold tables are optimized for BI and Direct Lake.
    • Collaboration — Engineers, analysts, and BI developers work at the right layer.
    • Reprocessing — Bronze allows full replay of ingestion.
    • Real‑Time — Event Streams → Bronze → Silver → Gold → Direct Lake.

    Medallion is not optional — it is the backbone of modern analytics.

    ← Back to Microsoft Fabric — Complete Learning Series

  • Medallion Architecture in Fabric — Pipelines, Notebooks & the Transformation Engine

    ← Back to Microsoft Fabric — Complete Learning Series

    Medallion Architecture in Fabric — The Perfect Fit

    Fabric’s unified platform makes Medallion architecture seamless. Every component fits naturally into the Bronze → Silver → Gold flow.

    • OneLake — All layers stored in one unified lake.
    • Delta Lake — ACID transactions, schema evolution, partitioning.
    • Lakehouses — Bronze, Silver, Gold stored as Delta tables.
    • Pipelines — Ingest raw data into Bronze.
    • Notebooks — Transform Bronze → Silver → Gold.
    • SQL Endpoint — Query Silver/Gold directly.
    • Direct Lake — Power BI reads Gold tables instantly.
    • Purview — Govern each layer centrally.

    Pipelines — The Ingestion Engine

    Fabric Pipelines handle enterprise‑grade ingestion into Bronze.

    Capabilities

    • Scheduled ingestion
    • Copy activities
    • Parameterization
    • Error handling & retry logic
    • Monitoring & logging
    • Notifications

    Common Pipeline Patterns

    • Full load
    • Incremental load
    • CDC load
    • Metadata‑driven load
    • Multi‑source ingestion

    Notebooks — The Transformation Engine

    PySpark notebooks transform Bronze → Silver → Gold.

    Common Transformation Patterns

    • Incremental loads
    • Merge operations
    • Partition pruning
    • Schema evolution
    • Delta optimization (Z‑Order, file compaction)

    Source → Pipeline → Bronze → Silver → Gold Flow

    A clean Medallion flow in Fabric looks like this:

    1. Ingest raw data into Bronze using Pipelines.
    2. Transform into Silver using PySpark notebooks.
    3. Model Gold tables for business consumption.
    4. Build semantic models using Direct Lake.
    5. Publish Power BI dashboards.

    Pipelines feed the Lakehouse. Notebooks shape the data. Gold delivers the value.

    ← Back to Microsoft Fabric — Complete Learning Series

  • Medallion Architecture — Direct Lake Integration & Real-Time BI

    Medallion + Direct Lake — Real‑Time BI

    Direct Lake reads Gold tables directly from OneLake — no refresh, no duplication, no latency.

    Benefits

    • No refresh cycles
    • No data duplication
    • Real‑time dashboards
    • Lower storage cost
    • Higher performance

    Requirements for Direct Lake

    • Gold tables must be Delta
    • Gold tables must be optimized
    • Gold tables must be partitioned
    • Gold tables must be business‑ready

    Medallion + Direct Lake = real‑time enterprise BI.

    Medallion Governance — Purview Integration

    Purview governs each layer differently:

    Bronze

    • Classified as raw
    • Lower sensitivity
    • Broad access

    Silver

    • Classified as cleaned
    • Medium sensitivity
    • Controlled access

    Gold

    • Classified as business‑critical
    • High sensitivity
    • Strict access

    Governance becomes structured and predictable across every layer.

  • Medallion Architecture — Dev/Test/Prod Strategy & Conclusion

    ← Back to Microsoft Fabric — Complete Learning Series

    Medallion Dev/Test/Prod Strategy

    Best Practices

    • Separate workspaces for Dev, Test, and Prod
    • Use deployment pipelines
    • Parameterize pipelines
    • Version control notebooks
    • Promote Gold tables carefully
    • Validate Silver transformations before promoting

    Medallion architecture thrives with a proper workspace strategy. Each environment mirrors the same Bronze → Silver → Gold structure, but with environment-specific data and permissions.

    End-to-End Architecture — How Everything Fits Together

    1. Ingest raw data into Bronze using Pipelines.
    2. Transform into Silver using PySpark notebooks.
    3. Model Gold tables for business consumption.
    4. Build semantic models using Direct Lake.
    5. Publish Power BI dashboards.
    6. Add real‑time streams for operational insights.
    7. Govern everything through Purview.
    8. Deploy across Dev/Test/Prod workspaces.
    9. Monitor performance and optimize Delta.
    10. Scale seamlessly as data grows.

    Conclusion — Medallion Is the Backbone of Fabric

    Medallion architecture is not just a pattern — it is the foundation of modern analytics. Fabric makes it natural, scalable, governed, and real‑time.

    • Bronze → Silver → Gold
    • Raw → Clean → Business‑Ready
    • Ingest → Transform → Model
    • Pipelines → Notebooks → SQL → Direct Lake

    This is the future of enterprise analytics.

    ← Back to Microsoft Fabric — Complete Learning Series

My journey from Datum to Data