Databricks for Practitioners : The Production Lakehouse Playbook

A practical, depth-first guide to running Databricks in production.The complete platform-and-data-engineering playbook for the engineers who own pipelines, govern catalogs, and keep workloads on schedule. Current to 2026, with examples on Azure Databricks and concepts that apply unchanged on AWS and GCP.Key Features• Build the governed Databricks platform layer by layer, from workspaces and compute to Unity Catalog, identity, and access control• Ship production data pipelines with Lakeflow Spark Declarative Pipelines, Jobs, Declarative Automation Bundles, and CI/CD from Git• Tune for cost and performance with Photon, Adaptive Query Execution, Liquid Clustering, and the Query Profile UIBook DescriptionMost Databricks books read like extended brochures. This one reads like a senior engineer sat next to you for a year. Databricks for Practitioners is the platform-and-data-engineering volume of Spark 4.0 from Scratch: sixteen chapters that take a fluent Apache Spark user and turn them into someone who can run Databricks at the depth a real organization demands.Part I builds the governed platform foundation: workspaces, classic and serverless compute, cluster policies, Unity Catalog from metastore to volume, privileges and ABAC, Governed Tags, lineage, Microsoft Entra ID identity, service principals, and managed Delta and Iceberg tables with UniForm.Part II covers the production data-engineering toolkit: ingestion with Lakeflow Connect and Auto Loader, bronze, silver, and gold pipelines with Lakeflow Spark Declarative Pipelines, scheduling with Lakeflow Jobs, deployment from Git via Declarative Automation Bundles and GitHub Actions, observability through system tables, and performance tuning with Photon and AQE.Examples run on Azure Databricks. What happens inside Databricks is identical on AWS and GCP; where the cloud seams differ (identity, storage, secrets, networking), chapters name the AWS and GCP equivalents explicitly.What you will learn• Architect a governed Databricks workspace from metastore to volume• Configure Unity Catalog with privileges, ABAC, Governed Tags, and lineage• Integrate Microsoft Entra ID, SCIM, and service principals• Build ingestion with Lakeflow Connect, Auto Loader, and streaming tables• Author bronze-silver-gold pipelines with Lakeflow Spark Declarative Pipelines• Deploy from Git with Declarative Automation Bundles and GitHub Actions• Observe billing, audit, query history, and lineage through system tables• Tune performance with Photon, AQE, Liquid Clustering, and the Query Profile UIWho this book is forData engineers, platform engineers, and architects who already know PySpark and now need to run it at production scale on Databricks. A working knowledge of PySpark, Spark SQL, and Delta Lake is expected. Readers new to Spark should start with Volumes 1 and 2 of the series.Table of Contents1. Databricks: The Platform on Top of Spark2. Workspaces, Notebooks, and Git Folders3. Compute: Classic, Serverless, and Cluster Policies4. Unity Catalog Architecture5. Access Control: Privileges, ABAC, and Governed Tags6. Identity: Entra ID, SCIM, and Service Principals7. Managed Tables: Delta, Iceberg, and UniForm8. Liquid Clustering and Predictive Optimization9. System Tables and Platform Observability10. Ingestion: Lakeflow Connect, Auto Loader, and Streaming Tables11. Lakeflow Spark Declarative Pipelines12. Lakeflow Jobs and Scheduling13. Declarative Automation Bundles14. CI/CD with GitHub Actions15. Performance: Photon, AQE, and the Query Profile UI16. Metric Views and the Bridge to Volume 4 Read more