← Back to Microsoft Fabric — Complete Learning Series
Governance for BI, Pipelines, Notebooks & Event Streams in Microsoft Fabric — Section 10 Conclusion
Section 10 has focused on one core idea: in Microsoft Fabric, governance is not an afterthought. It is designed into every workload, every workspace, and every data product. In this final article for the section, we look at how governance shows up specifically for Business Intelligence (BI), Data Factory pipelines, notebooks, and event streams — and then step back to see how these pieces form a single, unified governance fabric.
Governance for BI: Semantic Models and Direct Lake
Fabric builds on years of governance capabilities from Power BI and extends them across semantic models and Direct Lake datasets. The result is governed, trusted BI that operates at lakehouse scale without losing control.
Purview as the governance backbone for semantic models
Microsoft Purview is the central governance plane for Fabric, and semantic models are first-class citizens in that plane. When you connect Fabric to Purview, your BI assets become fully discoverable and governed:
- End-to-end lineage: Purview captures lineage from source systems (e.g., databases, SaaS apps) through lakehouses, warehouses, dataflows, and into semantic models and reports. This lets you answer questions like “Where does this KPI come from?” and “What will break if this source table changes?”
- Sensitivity labels: You can apply Microsoft Purview Information Protection labels (e.g., Confidential, Highly Confidential) to semantic models. These labels propagate to reports and even down to exported content, supporting consistent data protection policies.
- Access control: Purview works with Microsoft Entra ID (Azure AD) and Fabric workspaces so that access to semantic models is role-based and auditable. You can enforce who is allowed to see or query a model, not just view the report.
- Classification: Automated and manual classification rules can detect PII, financial data, or other regulated content inside the tables and measures of a semantic model. This is critical for identifying risk at scale.
- Audit logs: Every interaction with semantic models — access, modifications, refreshes — is logged. These logs flow into Microsoft Purview and Microsoft 365 Audit, enabling compliance, investigations, and operational monitoring.
Certified vs promoted datasets: creating a trust hierarchy
End users constantly ask, “Which dataset should I trust?” Governance for BI in Fabric relies heavily on clear endorsement signals: promoted and certified datasets.
- Promoted datasets
- Typically endorsed by power users, data stewards, or local BI teams.
- Signal that the dataset is generally useful and has some level of validation, but is not centrally guaranteed.
- Good for team-level or departmental solutions.
- Certified datasets
- Formally reviewed and approved by a central governance body — for example, a BI Center of Excellence, data governance council, or central data platform team.
- Require documented ownership, data definitions, refresh SLAs, and validation against master data or canonical sources.
- Are the official source of truth for organisation-wide reporting and executive dashboards.
In practice, you should define a clear process:
- Who can request certification for a dataset.
- What criteria must be met (data quality checks, documentation, lineage validation, SLA commitments).
- Who signs off (e.g., data owner + BI governance committee).
- How often certification is reviewed and renewed.
This trust hierarchy helps users quickly find the right data while giving governance teams a lever to promote the most reliable, well-managed data products.
Direct Lake: real-time analytics with full governance
Direct Lake is one of Fabric’s headline capabilities: Power BI semantic models can query data directly from OneLake in open formats like Delta, without import or duplication. From a governance perspective, this is a major advantage.
- Data never leaves OneLake: Instead of copying data into proprietary BI storage, you query the governed, central copy in OneLake. All the governance applied in the lake — security, labels, lineage, and access control — flows through to BI.
- Consistent policies: The same Purview policies, sensitivity labels, and RBAC rules used for lakehouses and warehouses apply to the tables that back Direct Lake models.
- Near real-time without extra copies: As new data lands in OneLake, Direct Lake models see it almost immediately. You get fresh insights without creating ungoverned data marts or shadow copies.
- Simplified compliance: Auditors and risk teams can see a single, governed path from source systems through OneLake to BI, rather than chasing copies across multiple BI servers and data marts.
Endorsement model: signalling trusted BI to the organisation
Endorsement is more than a badge on a dataset; it is a communication mechanism from governance teams to business users. A practical endorsement model in Fabric usually includes:
- Clear naming conventions for certified semantic models (for example,
ORG_Certified_Sales) so they stand out in search and in the dataset picker. - Metadata standards for descriptions, owner fields, and contact details so users know who to ask when questions arise.
- Documentation links from semantic models to wikis, glossaries, or data catalogs in Purview.
- Communication campaigns (townhalls, Teams announcements, internal portals) to educate users to prefer certified datasets whenever available.
These practices turn endorsement from a technical setting into a shared language for trust across the organisation.
Workspace-level BI governance
Workspaces are the primary container for BI content in Fabric — reports, semantic models, and related assets. Governance at the workspace level is about defining who can do what in each workspace.
- Who can publish
- Limit publishing capabilities (creating new reports or semantic models) to developers or power users with appropriate training.
- Use workspace roles (Admin, Member, Contributor, Viewer) to separate creation from consumption.
- Who can share
- Control who can share reports outside the workspace or outside the organisation.
- Use Fabric tenant settings to restrict external sharing, export to Excel, or data export where necessary.
- Who can modify
- Distinguish between users who can edit and those who can only view.
- Use separate workspaces for self-service and managed BI: for example, a centrally managed “Certified BI” workspace and departmental “Ad Hoc” workspaces.
By treating workspaces as governed products — with defined owners, roles, and guardrails — you avoid the chaos of unmanaged reports and datasets while still supporting self-service BI where it is appropriate.
Governance for Pipelines (Data Factory)
Data Factory pipelines in Fabric orchestrate the movement and transformation of data. Because they sit at the heart of data flows, strong governance over who can build, run, and change pipelines is essential.
Access control for pipelines
Pipelines are workspace-scoped assets, so workspace roles are the first layer of control. On top of that, you can define more detailed patterns:
- Creation rights: Allow only pipeline developers (e.g., data engineers) to create and delete pipelines. Avoid “everyone can create anything” in shared production workspaces.
- Edit rights: Separate edit rights from run rights. For example, you might allow operations teams to run or re-run pipelines but not modify activities or connections.
- Run permissions: Use run permissions and triggers carefully so that only authorised roles can execute pipelines that touch sensitive systems.
- View-only access: Provide view-only access to stakeholders who need to monitor status but should not change configurations.
Logging and monitoring as governance
Governed systems are observable systems. Fabric pipelines provide rich logging and monitoring, which you should treat as non-negotiable governance controls, not optional extras.
- Activity run history: Every pipeline activity run is logged with start time, end time, duration, inputs, and outputs. This supports troubleshooting and also shows exactly when sensitive operations occurred.
- Trigger history: For scheduled or event-based triggers, you can see when triggers fired, whether they succeeded, and which runs they initiated.
- Failure alerts: Integrate alerts with email, Teams, or ITSM tools so that failures on critical pipelines are immediately visible. Governance policies should define acceptable response times and escalation paths.
- Central log retention: Stream pipeline logs into a central log analytics solution (such as Azure Monitor / Log Analytics) with defined retention policies for compliance and forensic analysis.
Error handling and retry logic as governance controls
Error handling is often treated as an implementation detail, but it is a core part of governing data movement:
- Idempotent design: Pipelines should be designed so that retries do not produce duplicate data or corrupt downstream systems.
- Structured retries: Configure sensible retry counts and backoff strategies for transient failures (such as network issues) while failing fast on validation errors and schema mismatches.
- Clear failure paths: Route failed records or files into quarantine zones for later inspection rather than silently dropping or partially processing them.
- Governed error patterns: Document and standardise error handling patterns so that all pipelines behave consistently and are easier to support.
Version control via Git integration
Governed pipelines are versioned pipelines. Fabric’s Git integration with Azure DevOps and GitHub lets you treat pipelines as code:
- Change history: Every modification to a pipeline is captured in Git history, making it easy to see what changed, when, and by whom.
- Pull requests and reviews: Use PRs for changes to production pipelines. This introduces peer review, approval workflows, and automated checks before changes are merged.
- Rollback: If a change causes issues, you can roll back to a previous version quickly and confidently.
- Branching strategy: Adopt a branching model (e.g., main for production, develop for integration, feature branches for new work) that aligns with your wider DevOps practices.
Parameterisation to protect secrets and enable reuse
Hardcoding connection strings, secrets, or environment-specific values in pipelines is both a governance and security anti-pattern.
- Use parameters for environment-specific values: Connection endpoints, database names, file paths, and similar settings should be parameterised.
- Store secrets securely: Secrets should be stored in services like Azure Key Vault or managed identities, then referenced from pipelines, never hardcoded.
- Standard parameter sets: Define standard parameter names and conventions (e.g.,
env,region,targetLakehouse) to make pipelines portable and easier to govern.
Deployment pipelines for safe Dev → Test → Prod promotion
Fabric deployment pipelines provide a structured path from development to test to production. Using them for pipelines, not just BI content, is a powerful governance capability.
- Separated environments: Map each deployment stage to a workspace that represents Dev, Test, and Prod, with progressively stricter access controls.
- Configuration overrides: Use deployment rules and parameters to automatically adjust environment-specific settings (for example, test vs production data sources) without changing pipeline logic.
- Approval gates: Require approvals before promoting to production, ideally tied to change management processes.
- Consistent releases: Deploy related BI artefacts, pipelines, and other assets together, ensuring that data transformations and semantic models remain in sync.
Governance for Notebooks (Spark / Python)
Notebooks are powerful tools for data exploration, advanced analytics, and machine learning. In many organisations they are also a governance blind spot. Fabric helps bring notebooks into the governed mainstream.
Version control for notebooks
Notebooks should be treated as code, with the same governance standards as pipelines and applications.
- Git integration: Connect Fabric workspaces to Azure DevOps or GitHub repositories so that notebook changes are committed, reviewed, and versioned.
- Branching and PRs: Make notebook updates through feature branches and pull requests, not ad hoc changes in production.
- Tagged releases: Tag versions of notebooks that correspond to production deployments so you can reproduce results and analyses later if required by auditors.
Access control: view, edit, and run rights
Not all users should be able to run or modify all notebooks, especially those that touch sensitive data or production systems.
- View-only access: Analysts or auditors may need to inspect notebook logic without executing it.
- Run-only access: Operations roles might run production notebooks on a schedule or on demand but should not change the logic.
- Edit access: Restricted to data scientists, data engineers, or developers with appropriate training and accountability.
- Workspace and item-level permissions: Combine workspace roles with item-level permissions where necessary to fine-tune control over critical notebooks.
Code review and promotion processes
Unreviewed notebook code running in production can introduce data quality issues or security vulnerabilities. Establish a disciplined review process:
- Peer review: Require at least one peer review for any notebook that will run against production data or be used to generate production models.
- Standards and checklists: Define checklists covering performance, error handling, logging, and security considerations before approving changes.
- Separation of exploration and production: Keep exploratory notebooks in separate workspaces from production notebooks, and formalise promotion from one to the other through Git and deployment pipelines.
Audit trails: who ran what, when, and on which data
From a governance standpoint, you must be able to answer:
- Who executed a given notebook?
- When did they run it, and how long did it take?
- What data did it access, transform, or export?
Fabric integrates notebook execution with workspace logging and, in turn, with Purview and Microsoft 365 audit logs. Combined with lineage views across lakehouses and warehouses, this provides a traceable path from code execution to data changes.
Environment management: libraries and Spark pools
Uncontrolled environments are a common source of “works on my machine” issues. In a governed Fabric environment:
- Standardised library sets: Define approved libraries and versions for production use. Manage them centrally rather than letting each user install arbitrary packages.
- Managed Spark pool configurations: Standardise Spark pool sizes, autoscaling settings, and timeouts to balance performance, cost, and reliability.
- Environment-as-code: Where possible, describe environment configurations declaratively and track them in Git alongside notebooks.
Sensitive data handling in notebooks
Notebooks often provide deeper access to raw data than BI tools, which increases the governance stakes.
- No hardcoded credentials: Use managed identities, Azure Key Vault, or other secret management services. Credentials must never appear in clear text within notebook code or outputs.
- Respect sensitivity labels: When reading from labelled data in OneLake, assume that those labels apply downstream as well. Avoid exporting or visualising sensitive data outside approved channels.
- Minimise data extracts: Avoid exporting large volumes of sensitive data to local files, unmanaged storage, or external systems.
- Data masking and sampling: For development and testing, use masked or sampled datasets where possible rather than full production data.
Governance for Event Streams
Event streams bring real-time data into Fabric for analytics, monitoring, and operational decision-making. Because streaming data can include sensitive events (transactions, telemetry, user actions), it needs the same level of governance as batch data — with some streaming-specific twists.
Access control for event streams
As with other Fabric artefacts, event streams live in workspaces, so workspace roles are the starting point. However, event streams often touch external systems and mission-critical workloads, so more careful access policies are required:
- Creation and modification rights: Limit who can create or change event stream definitions, inputs, and outputs. Treat these permissions similarly to production pipelines.
- Connection management: Only trusted roles should be able to configure connections to external event sources and sinks, such as Event Hubs, Kafka, or operational systems.
- Operational roles: Define who is responsible for monitoring and managing running streams, including pausing, resuming, or scaling.
Lineage: tracking streaming data from source to destination
Purview lineage does not stop at batch jobs. Event streams and their connections to OneLake, warehouses, and downstream analytics are all part of the lineage graph.
- Source-to-sink visibility: For each event stream, you can trace which sources feed it and where enriched or transformed events land (for example, in Delta tables in OneLake).
- Impact analysis: If a source system changes its schema or event format, lineage helps you quickly identify which event streams and downstream analytics are affected.
- Regulatory traceability: For regulated industries, being able to show how real-time events flow into metrics and alerts is essential for compliance.
Sensitivity labels for streaming datasets in OneLake
Event streams often land data into tables or files in OneLake. Once in the lake, the same sensitivity labelling and classification mechanisms apply:
- Apply labels at rest: As streaming data is written to OneLake, ensure that the resulting tables or folders carry appropriate sensitivity labels.
- Propagate labels downstream: When these streaming-backed tables feed warehouses, semantic models, or notebooks, labels should propagate, preserving compliance context.
- Use labels for access decisions: Combine labels with access policies to restrict who can query highly sensitive streaming-derived data.
Monitoring: throughput, latency, and error rates
Streaming governance is also about ensuring that real-time data flows reliably and predictably:
- Throughput: Monitor volumes (events per second, MB/s) to detect bottlenecks, capacity shortfalls, or unexpected spikes that may indicate misuse or incidents.
- Latency: Track end-to-end latency from event ingestion to availability in OneLake or analytics dashboards. Governance policies may define maximum acceptable latency for critical use cases.
- Error rates: Monitor dropped events, parsing failures, or connectivity issues. High error rates can mask data loss that undermines trust in real-time metrics.
- Alerting and escalation: Define thresholds and alerts for key metrics, along with clear escalation procedures when they are breached.
Audit logs and Dev/Test/Prod promotion for event streams
As with other workloads, event streams should be auditable and follow a structured promotion path:
- Audit logs: Capture who created or modified event streams, changed connections, or altered routing rules. These logs support compliance, incident response, and operational reviews.
- Dev/Test/Prod environments: Use separate workspaces and deployment processes for event streams in development, test, and production. Avoid connecting development streams directly to production sources or sinks.
- Configuration-as-code: Where possible, store event stream definitions and routing configurations in Git, and promote them via deployment pipelines with approvals and automated checks.
Section 10 Conclusion — Governance Is the DNA of Fabric
Across BI, pipelines, notebooks, and event streams, a consistent pattern emerges: governance is built into Microsoft Fabric’s architecture. You are not bolting security and compliance on the side of an analytics platform; you are working within an analytics platform whose core design is governance-aware.
Purview at the centre of unified governance
Microsoft Purview provides the unified governance layer across every Fabric workload:
- Catalog and discovery for tables, semantic models, reports, pipelines, notebooks, and event streams.
- Lineage across batch and streaming, from source systems to final analytics artefacts.
- Classification and sensitivity labels that remain attached to data as it flows through the platform.
- Policy enforcement and audit that span both Fabric and connected services in the broader Microsoft data estate.
Workspaces and layered security
Workspaces are the organisational structure for governance in Fabric. Combined with security capabilities, they create multi-layered protection:
- RBAC (role-based access control): Workspace roles define who can create, edit, run, or view artefacts. Item-level permissions refine this further for sensitive assets.
- RLS (row-level security): Semantic models can enforce row-level filters so that users see only the subset of data they are entitled to.
- OLS (object-level security): Specific tables or columns can be hidden or restricted, even within shared models, enabling precise control over sensitive attributes.
These layers work together so that even if a user can access a report or model, they still only see the data appropriate for their role.
Lineage, labels, and deployment pipelines: visibility, compliance, and stability
Three other pillars complete the governance story:
- Lineage: Gives end-to-end visibility, enabling trust, impact analysis, and better collaboration between business and IT.
- Sensitivity labels: Encode compliance requirements into the data itself, ensuring that protection policies follow data across workloads and tools.
- Deployment pipelines: Provide a disciplined way to move artefacts from Dev to Test to Prod, reducing the risk of breaking changes, misconfigurations, or unreviewed code reaching production.
Together, these capabilities form what is arguably the most complete governance model ever built for analytics. Instead of stitching together separate governance tools for BI, data engineering, data science, and real-time analytics, Fabric gives you a single, coherent fabric of control.
Fabric as a unified governance platform
It is accurate to describe Microsoft Fabric as a unified analytics platform. But Section 10 has shown something deeper: Fabric is also a unified governance platform. Every workload — BI, pipelines, notebooks, event streams, lakehouses, warehouses — participates in the same patterns for identity, access control, lineage, labelling, monitoring, and DevOps.
This design means you can:
- Apply consistent governance policies across the entire analytics estate.
- Give business teams powerful self-service capabilities without surrendering control.
- Demonstrate compliance to regulators with clear evidence of controls and traceability.
- Reduce the operational burden of managing multiple disconnected platforms.
Why organisations that govern well will win
In a world where data volume, variety, and velocity are exploding, the organisations that win will not simply be those with the most data or the most sophisticated models. They will be the ones that can trust their data — to be accurate, secure, compliant, and understandable — at scale.
Microsoft Fabric gives you the platform to achieve that trust. By embracing governance as the DNA of your analytics — not a constraint, but a design principle — you enable more people to make better decisions, faster, with confidence. As you move beyond Section 10 into the rest of your Fabric journey, carry this mindset forward: every new workload, every new data product, and every new report is an opportunity to reinforce governance as a competitive advantage.
← Back to Microsoft Fabric — Complete Learning Series