DavidGadyan

Databricks Mind Map: Lakehouse Platform, Workspace, Clusters, Notebooks and DBFS

Data engineering foundations, Apache Spark, what the platform adds on top, the workspace, control and data planes, clusters, notebooks and DBFS: the Azure Databricks course as one connected map.

  1. Data engineering

    Why the platform exists

    What it is
    The course opens with what a data engineering team does: collect data from the operational systems of the business data owners, transform it, and facilitate its consumption by the data consumers. "Most data engineering projects are connecting these two teams with the right set of data at the right frequency at the right time."
    When to use it
    The frame for every tool on this map: each one serves ingestion, processing or serving, in batch or in streaming.

    1. Three groups of people

      Owners, engineers, consumers

      What it is
      From a Google Cloud diagram: business data owners run the operational systems (banking, e-commerce, OTT, IoT apps) that capture customer data; data consumers optimize, grow and monitor the business (fraud prevention, recommendation systems, sales and revenue reports); data engineers sit in the middle and connect the two with data.
      When to use it
      Requirements come from the consumers; the team works backwards from them to find which source systems hold the data.
      Figures from the course
      • The diagram classifies everyone who touches enterprise data into exactly three categories.Lesson: Introduction to Data Engineering

    2. Three functions

      Ingest, process, serve

      What it is
      Data integration or ingestion (collect from the sources and take the data into your own possession); data processing or transformation ("the most complex part of any data engineering project"); and data modelling and consumption (data marts, analytical stores and the connectivity the consumers need).
      When to use it
      Split any project into these three to decide which Databricks tool belongs where.

    3. Three approaches

      Batch, streaming, near real time

      What it is
      Batch processing, real-time stream processing and near-real-time stream processing. Batch collection runs at intervals (daily, hourly, every 30 minutes); streaming collects data continuously, as quickly as the source produces it; a project can mix both.
      When to use it
      Pick per source: some data arrives in batch, some in streaming, and the same architecture supports both.

    4. Lakehouse Medallion architecture

      Bronze, silver, gold

      What it is
      The reference architecture for a modern data engineering platform: ingest in batch or streaming with a tool suited to the source, then build the solution in layers. Bronze keeps the raw data as ingested, silver the processed high-quality data, gold the data modelled for consumption. Three layers is the minimum; all three processing approaches fit.
      When to use it
      The course implements it with Apache Spark and Databricks; the capstone project builds a lakehouse this way.
      Figures from the course
      • Minimum number of layers: three (bronze, silver, gold).Lesson: Introduction to Data Engineering

  2. Apache Spark

    The engine underneath

    What it is
    "An engine for executing data engineering, stream processing and machine learning on distributed clusters", offered through Spark SQL (ANSI SQL compliant), the DataFrame APIs for batch, Structured Streaming, graph processing and machine learning APIs, all on one framework.
    When to use it
    The course assumes you already know Spark SQL, the DataFrame API, Structured Streaming and Spark internals; Databricks is where that code runs.
    Figures from the course
    • Thousands of companies use Spark, including 80% of the Fortune 500.Lesson: Apache Spark to Data Engineering Platform

    1. The Spark stack

      Cluster manager, storage, APIs

      What it is
      Spark runs on a cluster through a resource manager (YARN, standalone, Kubernetes, Mesos) and reads or writes distributed storage (HDFS, S3, ADLS, Google Cloud Storage). On top sit the core RDD APIs in Scala, Java, Python and R, "not recommended for direct use", and above them Spark SQL and DataFrames, Structured Streaming, MLlib and GraphX.
      When to use it
      Write against the high-level APIs; the distributed execution underneath is hidden from you.

    2. Why Spark is popular

      Five reasons

      What it is
      Abstraction (simple SQL or DataFrame code runs in parallel without you noticing), ease of use (Spark SQL is the simplest interface), a unified platform (SQL, DataFrames, batch, streaming, ML and graph on one engine), open source, and a wide ecosystem of integrations built by the community.
      When to use it
      One skill set covers batch, streaming and ML instead of one tool per job.

    3. What Spark is not

      Five missing pieces

      What it is
      Open-source Spark has no storage layer (it needs HDFS, S3 or ADLS), no ACID transactions, no robust centralized metadata catalog (only "a small simple catalog"), no cluster management (it cannot create or destroy clusters) and no good automation APIs, SDKs or command-line tools. "Most popular big data engineering tool, but not complete in itself."
      When to use it
      The checklist a platform has to fill; each Databricks capability on this map answers one of these gaps.
      Figures from the course
      • Five critical capabilities missing from open-source Spark: storage, ACID, catalog, cluster management, automation.Lesson: Apache Spark to Data Engineering Platform

    4. Spark platforms

      Hadoop-based vs cloud-native

      What it is
      Cloudera Hadoop (the most popular on-premises platform), Amazon EMR, Azure HDInsight and Google Dataproc all run Hadoop behind the scenes with YARN as the cluster manager. Databricks is different: built on the cloud, for the cloud, purely Spark, no Hadoop cluster, no YARN, and not available on-premises.
      When to use it
      Choose Databricks when you want Spark without operating Hadoop; choose Cloudera when the workload must stay on-premises.

    5. Associate Developer syllabus

      What the Spark exam covers

      What it is
      The Databricks Certified Associate Developer for Apache Spark course (Python) walks through Spark architecture and transformations, DataFrame operations, advanced optimizations, Spark SQL, Structured Streaming and Spark ML, for data engineers, analysts and scientists with basic Python.
      When to use it
      The order to learn Spark in before the Azure Databricks course, which treats all of it as known.

      1. Architecture and transformations

        Components, hierarchy, actions

        What it is
        Spark components, the execution hierarchy, and the two kinds of operations: transformations and actions. The section explains "how Spark processes data differently from traditional systems".
        When to use it
        Understand this before tuning anything: it decides when work actually runs on the cluster.

      2. DataFrames

        View, filter, change columns

        What it is
        PySpark DataFrames: viewing, modifying and analysing data; filtering, column changes and built-in functions. "Hands-on skills for working with large datasets using PySpark."
        When to use it
        The everyday API for the silver layer: clean and reshape what landed in bronze.

      3. Optimizations

        Joins, AQE, Catalyst

        What it is
        Group-by operations, joins optimized with broadcast and shuffle strategies, and query execution improved with Adaptive Query Execution (AQE) and the Catalyst Optimizer, plus data partitioning.
        When to use it
        When a join or aggregation is slow: broadcast the small side, let AQE re-plan at runtime.

      4. Spark SQL

        Structured data, UDFs

        What it is
        Load, query and analyse structured data with Spark SQL and its built-in functions, connect it to other data sources, and build complex functions with user-defined functions.
        When to use it
        Analysts who know SQL work on the same engine as the Python code.

      5. Structured Streaming

        Continuous data flows

        What it is
        Real-time processing with Spark: streaming architectures built on Structured Streaming, including stream-stream and stream-static joins, "for responsive and scalable data solutions".
        When to use it
        The streaming side of the Medallion architecture: events land in bronze as they are produced.

      6. Spark ML

        Machine learning on the cluster

        What it is
        The certification course closes with "Machine Learning with Spark ML", and promises confidence "in data processing, streaming and machine learning with Python". MLlib is one of the APIs on the Spark stack.
        When to use it
        Train on the same cluster that prepared the data, without exporting it.
        AI angle
        Machine learning is one of the four capabilities Spark offers on a single framework, next to SQL, batch and streaming.

  3. Databricks platform

    What it adds to Spark

    What it is
    "A platform that brings technologies, tools and capabilities to develop and implement lakehouse solutions using Medallion architecture and other patterns": everything needed to design, develop, maintain and operate enterprise-grade lakehouse or data lake solutions.
    When to use it
    Each capability below fills one of the five gaps of open-source Spark, pre-integrated and ready to use.

    1. Spark on the cloud, Hadoop-free

      Optimized runtime

      What it is
      Spark was designed to run on Hadoop; Databricks makes it independent of Hadoop-native technology and integrates cloud storage securely. It also tunes "vanilla" Spark before shipping it, so "the same Apache Spark runs a little faster in Databricks".
      When to use it
      Keep your Spark code; drop the Hadoop cluster and the YARN tuning.

    2. Delta Lake and Delta tables

      ACID for the lake

      What it is
      Delta Lake is an open-source project that brings ACID transactions to distributed processing systems; Databricks integrates it with Spark as a pre-integrated, ready-to-use solution, and Delta tables are "one of the core features" of the platform. The syllabus continues with its optimization techniques and schema evolution, manual or automatic, in Delta Lake or during ingestion.
      When to use it
      Store every Medallion layer as Delta tables so writes are atomic and schemas can evolve.

    3. Unity Catalog

      Governance layer

      What it is
      "The data governance layer of your lakehouse platform": centralized metadata management, user management and security in one offering. The syllabus covers its purpose, configuration, user provisioning (bringing corporate users in) and securing tables and other data assets.
      When to use it
      The recommended way to set up the storage layer: DBFS mounts give everyone in the workspace the same access, Unity Catalog gives fine-grained control.

    4. Cluster management

      Launch, scale, release

      What it is
      Launch clusters, autoscale them up or down and release them from inside the platform, "without directly going and working with your cloud solutions". Open-source Spark cannot create, destroy or manage clusters at all.
      When to use it
      The Compute menu replaces the cloud console for day-to-day infrastructure; details in the Clusters branch.

    5. Photon

      Transparent acceleration

      What it is
      A query accelerator Databricks created: "you just switch it on and it starts giving a boost to your Spark SQL queries and DataFrame APIs". A new accelerated query engine for Spark, chosen per cluster, with extra cost; the course turns it off for learning.
      When to use it
      Production SQL and DataFrame workloads where speed is worth the surcharge; not for learning clusters.
      Figures from the course
      • Photon is a checkbox in the cluster form; it "comes with some extra cost".Lesson: How to create Spark Cluster

    6. Notebooks and workspace

      The IDE of the platform

      What it is
      "Kind of an IDE" for the day-to-day project activities: development, testing and integration with source control. Details in the Workspace and Notebooks branches.
      When to use it
      Write, run and version the code in the browser, attached to a cluster.

    7. Administrative controls

      Users, groups, policies

      What it is
      A robust security scheme: user groups, policies, and access and activities controlled by those policies. In Azure the workspace comes in two pricing tiers; the main difference is that premium includes role-based access control, "and a few other features".
      When to use it
      The course picks premium "because we want to see everything"; standard is cheaper.
      Figures from the course
      • Two pricing tiers: standard and premium; premium adds role-based access control.Lesson: Creating Databricks Workspace Service

    8. Automation: APIs, CLI, Terraform

      Project automation

      What it is
      APIs, SDKs and command-line tools to automate project activities, plus Terraform integration. The syllabus dedicates a section to the Databricks REST API and CLI "so you understand how project-related automation can be performed".
      When to use it
      Create workspaces, clusters and jobs from scripts and CI/CD instead of clicking through the UI.

    9. Three clouds, one platform

      Azure, AWS, GCP

      What it is
      Available on Azure, AWS and Google Cloud: "the same capability, same solution, on a choice of cloud platforms". Differences are infrastructure, setup and configuration, mostly transparent to the project team.
      When to use it
      Choose the cloud your organization is already aligned or partnered with; the course uses Azure.

    10. Lakehouse toolset

      The rest of the syllabus

      What it is
      The tools the course outline goes on to teach for building the lakehouse: incremental ingestion with Autoloader, Delta Live Tables, Workflows, Repos and the capstone project. Only the outline's own descriptions are on these cards.
      When to use it
      The order of work after storage and compute exist: ingest, transform declaratively, orchestrate, version, deliver.

      1. Incremental ingestion and Autoloader

        From sources into the lake

        What it is
        Taught "from the architecture point of view": how to architect ingestion from source systems into the lakehouse, which tools and capabilities Databricks offers for incremental ingestion, how to manage schema evolution during ingestion, and what the preferred tool is. Databricks Autoloader gets its own lesson.
        When to use it
        New files keep arriving in cloud storage and only the new ones should land in bronze.

      2. Delta Live Tables

        Declarative ETL

        What it is
        "The declarative ETL offering of the Databricks platform": DLT pipelines. In the workspace it has its own menu entry, which opens the Delta Live Tables tab of the Workflows page.
        When to use it
        Describe the tables you want and let the pipeline manage the dependencies and refreshes.

      3. Workflows

        Jobs and dependencies

        What it is
        Orchestrate jobs, create job dependencies and build pipelines: "complex workflows where you can schedule some tasks and create dependencies between tasks". The Job runs menu entry opens the same page.
        When to use it
        Run the bronze, silver and gold notebooks in order, on a schedule, with retries you do not hand-roll.

      4. Repos

        Source control in the workspace

        What it is
        Connect the workspace to GitHub, GitHub Enterprise, Azure DevOps and other source-control providers; files live in the repository, you open and edit them from the workspace and commit from there.
        When to use it
        Team development with reviews and history instead of notebooks living only in a user folder.

      5. Capstone: Medallion with CI/CD

        Requirement to deployment

        What it is
        A bonus project that starts from a business requirement and walks through solution design, the design decisions and the implementation of a lakehouse with the Medallion architecture, then a testing strategy, test automation and a CI/CD pipeline built with Azure DevOps.
        When to use it
        The shape of a real engagement: design decisions documented, tests automated, deployments through a pipeline.

  4. Workspace

    Gateway to the platform

    What it is
    "Your gateway to the Databricks platform": everything is done from the web UI of the workspace (or through the REST API and CLI). In Azure it is created as an Azure Databricks workspace, "a serverless application", and opened from the portal's Launch workspace button or its own URL with single sign-on.
    When to use it
    One workspace per project, or several per project: development, QA and production.

    1. Azure account and portal

      Free credits, resource groups

      What it is
      A free Azure account gives USD 200 of credit for 30 days, popular services free for 12 months and 40+ always-free services; nothing is charged unless you upgrade to pay-as-you-go by hand. The portal is the web UI (search box, Cloud Shell, Entra ID tenant); a resource group is a named collection of resources that share lifecycle and permissions and can be deleted in one shot.
      When to use it
      Put everything for one project or course in one resource group so clean-up is a single delete.
      Figures from the course
      • Free account: USD 200 of credit, valid 30 days.Lesson: Creating Azure Cloud Account
      • A free subscription allows 4 CPU cores at a time, so only single-node clusters.Lesson: How to create Spark Cluster

    2. Creating the workspace

      Subscription, group, tier

      What it is
      In the Azure Databricks service: choose the subscription (the billing account), a resource group, a workspace name, a region (the instructor picks US East as the cheapest) and the pricing tier, then Review + create. Deployment takes a few minutes; the workspace is cheap to keep, so create one and leave it for the duration of the course.
      When to use it
      The first thing to do on a new project, before clusters or data.
      Figures from the course
      • Deployment of the workspace took about 5 to 10 minutes in the demo.Lesson: Creating Databricks Workspace Service

    3. The left menu

      Where the work happens

      What it is
      Most functionality sits in the left menu: Workspace, Recent, Catalog, Workflows and Compute at the top; then Job runs, Data ingestion and Delta Live Tables; then the SQL and Machine Learning sections; then Marketplace and Partner Connect. The top-right menu holds user settings, admin settings and the account console.
      When to use it
      Keep it expanded while learning; it collapses to icons otherwise.

      1. Workspace folders

        Shared, Users, Repos

        What it is
        The Workspace entry is the gateway to code: a Shared folder, a Users folder with each person's home, and Repos. In your home you create folders, notebooks, files, queries, alerts, dashboards and ML experiments, and import notebooks (.dbc archives) with the Import action.
        When to use it
        Where the development team logs in to write, run and test code.

      2. Catalog explorer

        Metadata and objects

        What it is
        The Catalog entry opens Catalog Explorer: metadata and objects such as databases, tables, views and functions. Empty on a new workspace; it fills as the course progresses.
        When to use it
        Browse what exists before writing a query against it.

      3. Compute menu

        Clusters without the console

        What it is
        Create, terminate and manage clusters from the workspace itself: all-purpose clusters, job compute, SQL warehouses, cluster pools and cluster policies. "You need not go back to your Azure portal and create infrastructure."
        When to use it
        Any notebook needs a running cluster from here first; see the Clusters branch for the form.

      4. Data ingestion page

        Native and partner connectors

        What it is
        A page listing native integrations, built into the platform with Spark, to ingest from various sources, plus partner tools for the sources those do not cover.
        When to use it
        Pick a suitable connector or partner tool to bring a source system into the lakehouse.

      5. SQL section

        Serverless warehouse

        What it is
        "Databricks is also a serverless on-cloud data warehouse": the SQL section has serverless SQL warehouses, the SQL editor, queries, dashboards, alerts and query history.
        When to use it
        BI and ad-hoc SQL on the gold layer without a Spark cluster of your own.

      6. Machine Learning section

        Experiments and models

        What it is
        "Databricks is also a platform for implementing machine learning algorithms"; those functionalities live under the Machine Learning menu. Notebooks have an Experiments panel "mainly used by data scientists", and Partner Connect lists machine-learning partners.
        When to use it
        The data scientists' side of the same workspace: experiments next to the engineers' pipelines.
        AI angle
        The course names the ML menu and the Experiments panel but, being a data engineering course, teaches neither.

      7. Marketplace and Partner Connect

        Data and partners

        What it is
        Marketplace: data that other organizations sell or give away, powered by data sharing. Partner Connect: partner vendors by function (ingestion, visualization, data preparation, machine learning, security, governance) with easy integration into the platform.
        When to use it
        Enrich with an external dataset, or plug in a vendor tool the platform does not ship itself.

  5. Control plane and data plane

    Who owns what

    What it is
    The Azure Databricks workspace is deployed in two planes. The control plane is the Databricks software, a serverless application in Databricks' control; the data plane is everything created in the customer's Azure subscription, under the customer's control and on the customer's bill.
    When to use it
    The takeaway the course insists on: clusters and data are always in the data plane, configurations in the control plane.

    1. Control plane

      The Databricks application

      What it is
      Holds the web application, the notebooks created in the workspace, and the configuration of jobs, queries, alerts, workflows and clusters. No back-end access: the only ways in are the web UI and the REST APIs or command-line tools.
      When to use it
      Automation scripts talk to this plane: list clusters, start jobs, export notebooks.

    2. Data plane

      Your subscription, your bill

      What it is
      Cluster virtual machines are created in the customer's Azure subscription: you can see them, log into them and terminate them, and you pay Azure for them. The data storage layer also lives here.
      When to use it
      Cost control happens here: a terminated cluster still leaves disks and a public IP until it is deleted.

    3. Workspace root storage

      Auto-created, not for data

      What it is
      Every workspace creates one default storage account in the data plane, exposed as the DBFS root. It is secured by the workspace and awkward to reach from the cloud account directly, and "not the storage layer which we should be using" for the lake.
      When to use it
      Leave it to Databricks; put project data in a storage account you own.

    4. Your own data storage layer

      Full control and custody

      What it is
      In a typical project a separate storage account is created in your subscription "so that we have full control of it, we keep data in our own custody and we secure it in the way we want".
      When to use it
      The lakehouse storage: ADLS Gen2 mounted or, better, governed through Unity Catalog.

    5. Ways in

      UI, REST API, CLI, SSO

      What it is
      Users sit outside the control plane and reach it through the web UI, authenticated by single sign-on with their Azure credentials, or through REST-based API clients and command-line tools.
      When to use it
      Humans use the UI; pipelines and CI/CD use the API and CLI.

  6. Clusters

    Where Spark code runs

    What it is
    "Spark code runs only on a Spark cluster." An all-purpose cluster is created from the Compute menu with a name, a cluster type, an access mode, a Databricks runtime, worker and driver VM sizes, autoscaling and auto-termination; the form shows the estimated cost in DBU per hour.
    When to use it
    Create a small learning cluster; create a multi-node one only when the lesson needs it.
    Figures from the course
    • Starting a cluster took up to 5 to 7 minutes in the demo.Lesson: How to create Spark Cluster

    1. Cluster types

      All-purpose, job, SQL warehouse

      What it is
      All-purpose clusters for interactive work, job clusters for scheduled jobs, SQL warehouses for the SQL section, plus cluster pools and cluster security and access policies. Creating a cluster from a notebook offers the Personal Compute policy.
      When to use it
      Notebooks on all-purpose clusters while developing; the same code on job clusters in Workflows.

    2. Multi-node vs single node

      Driver and executors

      What it is
      A multi-node cluster starts the driver and the executors on separate machines; a single-node cluster runs both on one machine and has no autoscaling. A free Azure account allows 4 CPU cores, so the demo multi-node configuration (two 4-core workers plus a driver, 12 cores) cannot be created there.
      When to use it
      Single node is "more than enough for learning and development"; multi-node when the concept needs real parallelism.
      Figures from the course
      • Two workers plus one driver at 4 cores each = 12 cores; the free account caps at 4.Lesson: How to create Spark Cluster

    3. Access modes

      Single user, shared, no isolation

      What it is
      Three options in the form: single user, shared, and no isolation shared. The course picks single user and defers the details to later chapters.
      When to use it
      Single user for your own development cluster.

    4. Runtime and Photon

      LTS by default

      What it is
      The default Databricks runtime is the latest with long-term support; Photon acceleration is a separate checkbox with extra cost, unticked in the course. The runtime version can be changed later from the cluster's Edit button.
      When to use it
      Stay on the default LTS unless a library needs something newer.

    5. Worker and driver sizing

      Per-node VM

      What it is
      The worker node type is a VM configuration per worker; the default selection may be larger and more expensive than you need. The driver can match the worker size or differ. "Do not forget to check two things: the node type and your DBU cost."
      When to use it
      Learning: the smallest VM for both; production: size the workers to the data, the driver to the collect.
      Figures from the course
      • Minimum VM: 14 GB of memory and 4 CPU cores; two such workers give 8 cores and 28 GB.Lesson: How to create Spark Cluster

    6. Autoscaling

      Min and max workers

      What it is
      Enabled by default with a minimum and maximum number of workers; Databricks adds workers as the workload grows and terminates them as it falls. The course sets 1 to 2 for learning.
      When to use it
      Variable workloads: pay for the second worker only while the job needs it.
      Figures from the course
      • Default autoscaling range: 2 to 8 workers; the demo uses 1 to 2.Lesson: How to create Spark Cluster

    7. Auto-termination, terminate, delete

      Protect against idle cost

      What it is
      Auto-termination is on by default at 120 idle minutes and can be set to 10 or more; "you must set it to protect against unnecessary costs". A terminated cluster can be restarted, but its disks, network interfaces and public IP stay in the resource list and Azure keeps charging a small amount for them until the cluster is deleted permanently.
      When to use it
      Learning: 30 minutes or less, and delete the cluster after practice.
      Figures from the course
      • Default auto-termination: 120 minutes idle; the course recommends 30 for learning clusters.Lesson: How to create Spark Cluster
      • Azure may take up to 10 minutes to clean up the resources after a permanent delete.Lesson: How to create Spark Cluster

    8. Advanced options

      Tags, config, logs, init scripts

      What it is
      Custom tags (user-defined metadata for reports), default Spark configurations and environment variables applied before launch, an optional cluster log location (logs are not collected by default), and init scripts: shell scripts that install or configure something on the VM before Spark starts.
      When to use it
      Databricks sets good defaults; come here for a library every node needs or a config every job shares.

    9. DBU cost

      Databricks Unit per hour

      What it is
      The DBU, Databricks Unit, is the indicator of Databricks' own charge; the summary shows a range per hour for the chosen configuration. The consolidated Azure bill adds the VM and other resource costs on top. The instructor's rule of thumb: about one US dollar per DBU, often less.
      When to use it
      Estimate a job: DBU per hour times hours, plus the VM hours.
      Figures from the course
      • The demo configuration showed 1 to 3 DBU per hour.Lesson: How to create Spark Cluster

    10. Cluster page tabs

      Libraries, logs, Spark UI

      What it is
      Once started: Notebooks attached, Libraries to install, Event log (start and stop entries), Spark UI, Driver logs (stdout, stderr, log4j) and Metrics with the preconfigured Ganglia monitoring; Databricks was also adding web terminals and RStudio. Edit changes the configuration; some changes restart the cluster.
      When to use it
      A slow notebook: Spark UI for the stages, driver logs for the errors, metrics for memory.

  7. Notebooks

    The development IDE

    What it is
    A notebook is your code file, written in cells and run on an attached cluster; the workspace returns each cell's output below it. Created inside a workspace folder with a default language; the course uses Python with mixed-language cells.
    When to use it
    All the course code lives in exercise notebooks (blank, with step-by-step questions) and solution notebooks (filled in).

    1. Languages, cells and state

      Python, SQL, Scala, R

      What it is
      Four default languages: Python, SQL, Scala and R. Code goes in cells; Ctrl+Enter runs the current cell, Shift+Enter runs it and moves on, Escape leaves edit mode for command mode. The notebook keeps the state of executed cells, so a variable from cell one is visible in cell two until you clear the state.
      When to use it
      A NameError after reopening a notebook usually means the defining cell has not run yet.
      In code
      Ctrl + Enter    run the current cell
Shift + Enter   run the cell and move to the next one
Esc             leave the cell: command mode

    2. Attach, detach, reattach

      One notebook, many clusters

      What it is
      A notebook runs only when attached to a running cluster (the Connect button lists them or creates one). From the cluster dropdown you can detach and attach to another cluster, restart the cluster, open the Spark UI or the driver logs.
      When to use it
      Detach from the dev cluster, attach to the test cluster, and run the same notebook in both environments.

    3. File, Edit, View, Run menus

      Export, format, run all

      What it is
      File: new, clone, rename, move, delete and Export in four formats (Databricks archive .dbc, Python source, IPython notebook, HTML); the archive re-imports into any workspace. Edit: cut, copy, paste, find and replace, format cell or notebook, Python indentation (default four spaces). View: standard, results only, side by side. Run: all, cell, all above, all below, selected text; clear outputs and clear state.
      When to use it
      Export as .dbc to move a notebook between workspaces; "run all" to test the whole file.
      Figures from the course
      • Four export formats: DBC archive, Python source, IPython notebook, HTML.Lesson: Working with Databricks Notebook

    4. Right-side panel

      Comments, history, libraries

      What it is
      Comments on selected lines for review in shared notebooks; Experiments (for data scientists); revision history, since every save creates a version you can restore; a variable explorer for debugging; and Python libraries at three levels: notebook (only pip preinstalled), cluster (nothing extra) and runtime (many preinstalled). Install extras with pip in the notebook or at cluster level for every notebook on it.
      When to use it
      Restore yesterday's version after a bad edit; check the runtime before pip-installing a library it already has.

    5. Magic commands

      Change what a cell does

      What it is
      A line starting with % at the top of a cell changes the cell's language or hands it to the file system or the shell. Databricks notebooks also inherit the IPython magics. "You can create a sequence of code blocks in multiple languages and run it as a single notebook."
      When to use it
      SQL for the query, Python for the loop, Markdown for the explanation, all in one file.

      1. %sql, %scala, %python

        Mix languages per cell

        What it is
        Each cell defaults to the notebook's language; %sql, %scala or %python switches that one cell. %python is only needed when the notebook's default is Scala or SQL.
        When to use it
        A quick SQL check in the middle of a PySpark notebook.
        In code
        %sql
SELECT 1 + 1
%scala
println("hello")

      2. %md

        Documentation cells

        What it is
        Turns a cell into Markdown for notes and documentation: links, images and rich formatting. The course recommends learning Markdown to "create excellent documentation within your notebooks".
        When to use it
        A heading and a paragraph above each step of a pipeline notebook.
        In code
        %md
## Step 1: load the raw files

      3. %fs

        File-system commands on DBFS

        What it is
        Runs common file-system commands over the distributed file system, such as ls on the preloaded datasets; it uses the dbutils file-system utility behind the scenes, but does not offer the full set of Linux commands.
        When to use it
        Look at what landed in a directory before reading it with Spark.
        In code
        %fs ls
%fs ls /databricks-datasets/airlines

      4. %sh

        Linux shell on the driver

        What it is
        Runs Linux shell commands, covering what %fs lacks: the course shows tail, unavailable under %fs, working under %sh.
        When to use it
        Inspect a local file or install something on the driver with a one-liner.
        In code
        %sh tail /path/to/a/file

      5. %lsmagic, ? and ??

        List, document, read source

        What it is
        %lsmagic lists every available magic, most inherited from IPython. Append ? to a magic for its documentation and ?? for its source code; the course shows that %pip is a Python function running the pip system command.
        When to use it
        Find out what a magic does without leaving the notebook.
        In code
        %lsmagic
%pip?
%pip??

    6. Databricks Utilities (dbutils)

      Utilities from Python code

      What it is
      A package of utilities: credentials, data, fs, jobs, library, meta, notebook, preview, secrets and widgets. Many are experimental or in preview; fs, notebook and widgets are "the most commonly used". Every utility documents itself with help().
      When to use it
      Learn the mechanics of one and the others follow the same pattern.
      In code
      dbutils.help()
dbutils.fs.help()
dbutils.fs.help("cp")

      1. dbutils.fs

        DBFS from Python

        What it is
        The same operations as %fs (cp, head, ls, mkdirs, mv, put, rm, mount), behaving like the Linux commands, but callable from Python so you can loop over results and "leverage the power and flexibility of a programming language".
        When to use it
        List a directory and process each file name in a loop.
        In code
        for f in dbutils.fs.ls("/"):
    print(f.name)

      2. dbutils.notebook

        Call one notebook from another

        What it is
        Only two methods, run and exit, but together "they allow you to design a modular Spark application": a main notebook calls child notebooks (full path, timeout in seconds, optional arguments) and reads the exit value each one returns; the output links to the child notebook job.
        When to use it
        One notebook per concern, reused from a main notebook, instead of one giant file.
        In code
        result = dbutils.notebook.run("/Users/me/child_notebook", 60, {"name": "value"})
dbutils.notebook.exit("done")

      3. dbutils.widgets

        Notebook parameters

        What it is
        Adds input arguments to a notebook: text (the most common), combobox, dropdown and multiselect, each with a name, a default and a label shown at the top of the notebook; get reads the value, removeAll clears them. The parent notebook passes values as key-value pairs in run.
        When to use it
        Parameterized code: the same notebook runs for a different date or source depending on the input.
        In code
        dbutils.widgets.text("name", "", "Your name")
value = dbutils.widgets.get("name")
dbutils.widgets.removeAll()

      4. The other utilities

        secrets, jobs, data, library

        What it is
        credentials shows and assumes user roles; data summarizes a DataFrame; jobs interacts with workflow jobs; library compiles applications that use dbutils locally; meta hooks into the compiler; preview exposes hidden utilities; secrets reads secure credentials, the fix for secrets hard-coded in a mount notebook.
        When to use it
        Reach for secrets as soon as a notebook needs a client secret.
        In code
        dbutils.secrets.help()

  8. DBFS and storage

    Cloud storage as a directory

    What it is
    The Databricks File System: "a distributed file system abstraction on top of scalable cloud storage, integrated into the workspace and available on clusters". In simple terms, a virtual file system that lets you treat cloud storage as directories, subdirectories and files.
    When to use it
    How notebooks address data: paths instead of storage-account APIs.

    1. DBFS root

      dbfs:/ is the default

      What it is
      The storage account Databricks creates per workspace (named after the resource group and workspace) is mounted as the DBFS root. The dbfs: qualifier is optional because DBFS is the default file system; the root holds Databricks' own configuration directories and /databricks-datasets, sample data available to every workspace. Not for your project data.
      When to use it
      Browse the sample datasets to practise; mount your own storage for anything real.
      In code
      %fs ls dbfs:/
%fs ls /databricks-datasets/
%fs mounts

    2. Driver local file system

      file:/

      What it is
      The other file system available by default: file: addresses the local disk of the driver machine (lib64, boot and the rest of a Linux root).
      When to use it
      Temporary files that a single-machine library writes before you copy them to DBFS.
      In code
      %fs ls file:/

    3. Mounting ADLS Gen2

      Five steps

      What it is
      Bring a storage account you own under the DBFS root: create an ADLS Gen2 storage account, create a container, create a service principal with a secret, grant it access to the account, then mount the container from the workspace. Four steps happen in Azure, the last one in a notebook.
      When to use it
      The course's way to get the sample data into the platform; see the limits card before using it for a project.
      Figures from the course
      • User mount points always live under /mnt in the DBFS root.Lesson: Mounting ADLS to DBFS

      1. 1. Storage account

        ADLS Gen2 = hierarchical namespace

        What it is
        Create a storage account in the same region as the workspace with a globally unique name (no hyphens), standard performance, locally redundant storage for learning, and Enable hierarchical namespace ticked under Advanced: that makes it Azure Data Lake Storage Gen2, "specifically designed and optimized for data lake storage". Gen1 and plain blob storage are not recommended.
        When to use it
        The data storage layer of the project, kept out of the workspace root storage.

      2. 2. Container

        The root directory

        What it is
        A container is the root directory inside the account (the account itself only carries configuration such as tier and redundancy); create as many as you need. From the portal you can create subdirectories and upload files into it.
        When to use it
        One container per data area; the course uses one for the sample data.

      3. 3. Service principal and secret

        A user for the app

        What it is
        An application user: register an app in Microsoft Entra ID (the renamed Azure Active Directory) and note its client ID and tenant ID, then create a client secret under Certificates & secrets. The secret value is shown only once; copy it.
        When to use it
        Databricks authenticates to the storage as this principal, not as a person.
        Figures from the course
        • The demo secret was created with three months of validity.Lesson: Mounting ADLS to DBFS

      4. 4. Role assignment

        Storage Blob Data Contributor

        What it is
        On the storage account, Access control (IAM), Add role assignment: the predefined Storage Blob Data Contributor role, assigned to the service principal, lets it create directories and files and read, write and delete.
        When to use it
        The permission step people forget; without it the mount call fails.

      5. 5. dbutils.fs.mount

        OAuth configs and abfss://

        What it is
        Copied from the documentation: a configs dictionary with the OAuth auth type, Microsoft's provider class, the client ID, the client secret and the token endpoint with the tenant ID, then dbutils.fs.mount with the abfss source (container and storage account) and a mount point under /mnt. Returns True and the data appears under the mount.
        When to use it
        Everyone with access to the workspace can then read /mnt/files like a local folder.
        In code
        configs = {"fs.azure.account.auth.type": "OAuth",
  "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
  "fs.azure.account.oauth2.client.id": client_id,
  "fs.azure.account.oauth2.client.secret": client_secret,
  "fs.azure.account.oauth2.client.endpoint": f"https://login.microsoftonline.com/{tenant_id}/oauth2/token"}
dbutils.fs.mount(source=f"abfss://{container}@{storage_account}.dfs.core.windows.net/",
                 mount_point=f"/mnt/{mount_name}", extra_configs=configs)
%fs ls /mnt/files

    4. Unmount and remount

      The data stays put

      What it is
      dbutils.fs.unmount removes the mount point; reading it afterwards raises "no such file or directory", but nothing is lost: the files are still in the container and reappear when you mount again. Files created from the portal and from the workspace land in the same location.
      When to use it
      Rotate a secret: unmount, update the configs, mount again.
      In code
      dbutils.fs.unmount("/mnt/files")

    5. Two problems with mounts

      Why Unity Catalog instead

      What it is
      First, the notebook exposes the client ID, tenant ID and secret to anyone who reads the source control; the fix is Azure Key Vault read through dbutils.secrets. Second, a mounted directory is visible to every user of the workspace with no easy fine-grained control. "DBFS is not a recommended approach for setting up your storage layer"; the recommended approach is Unity Catalog.
      When to use it
      Mounts for learning and for data everyone may see; Unity Catalog for anything with access rules.

Sources: Coursera, "Fundamentals of Azure Databricks" by Packt (the Mastering Azure Databricks for Data Engineers videos) and the section overviews of "Databricks Certified Associate Developer for Apache Spark using Python" by Packt, transcripts pulled 2026-11-05. Definitions, commands and figures are the courses' own; Unity Catalog, Delta Lake, DLT, Workflows and Repos appear only as the course outline describes them. Check docs.databricks.com for current UI, limits and pricing.