Designing the Bronze Layer — Raw Data

Purpose

Capture raw data exactly as it arrives.

Characteristics

• Minimal transformation
• Raw schema
• Raw files or raw Delta tables
• High volume
• Append‑only

Best Practices

• Store raw files in /Files/Bronze
• Convert to Delta for consistency
• Add ingestion metadata (timestamp, source)
• Avoid business logic
• Use Pipelines for ingestion
• Use Event Streams for real‑time

Common Bronze Sources

• APIs
• Databases
• ERP systems
• CRM systems
• IoT devices
• Logs
• S3/ADLS shortcuts

Bronze is your “source of truth.”

Designing the Silver Layer — Cleaned & Conformed

Purpose

Transform raw data into clean, standardized, analytics‑ready tables.

Best Practices

• Use PySpark notebooks
• Use Delta Lake merges
• Apply schema enforcement
• Remove duplicates
• Standardize column names
• Add surrogate keys
• Partition large tables

Common Silver Tasks

• Remove nulls
• Standardize date formats
• Convert strings to numeric types
• Join reference tables
• Apply business logic
• Flatten nested JSON

Designing the Gold Layer — Business‑Ready

Purpose

Provide BI‑optimized, business‑friendly tables.

Best Practices

• Use SQL endpoint for modeling
• Build fact + dimension tables
• Use surrogate keys
• Use incremental loads
• Partition fact tables
• Use aggregation tables

Common Gold Tables

• FactSales
• DimCustomer
• DimProduct
• FactInventory
• FactFinance
• KPI tables

Gold is your “analytics layer.”