The data glossary, written for the person paying for it.

Decides how the data platform should be built, before anyone builds it. Chooses the technologies and architecture patterns, defines how data is stored, processed and integrated, and sets the standards on scalability, security and governance that the engineering team then follows.

Not the same as a data engineerAn architect works at platform level and mostly produces decisions and designs. An engineer works at system level and produces things that run. On a small project one person does both; on a corporate platform, treating them as the same job is how you end up with three incompatible pipelines.

Data architecture service →

Builds and maintains the infrastructure that moves, transforms, stores and processes data. The job is to get data out of the systems where it lives — ERP, CRM, internal apps, databases — and into the place where it will actually be used, reliably and on schedule.

Typical stack: Python, SQL, Spark, Databricks, Kafka, Airflow, dbt, and one of the three big clouds.

Data engineering service →

Turns available data into something a business can decide with: KPIs, dashboards, reports, reporting-oriented data models. Sits closest to the business user of the three roles — the job is not to build the platform but to make the numbers legible and defensible.

Typical stack: Power BI, Tableau, Looker, SQL, Excel, Microsoft Fabric.

Business intelligence service →

A platform for processing, engineering and analysing large volumes of data, built around Apache Spark. Different profiles — engineers, analysts, data scientists — work on the same platform to transform data, run jobs, train machine learning models and build data products.

It is not tied to one cloud: it runs on Azure, AWS and Google Cloud alike.

Not the same as SnowflakeDatabricks leans towards processing and engineering; Snowflake leans towards storing and querying structured data for analytics. They compete in the middle and plenty of companies run both. Anyone who tells you they are interchangeable is selling one of them.

A Databricks warehouse on SAP data →

The open-source engine for distributed data processing that sits underneath a large part of the modern data stack, Databricks included. When a job is too big for one machine, Spark splits it across many. PySpark is its Python interface — the one most data engineers actually write.

PySpark in a real-time telco pipeline →

A cloud platform specialised in storing, processing and querying large amounts of data for analytics, reporting, BI and data science. Like Databricks, it is independent of any single cloud: it runs on top of AWS, Azure or Google Cloud infrastructure rather than belonging to any of them.

Google Cloud’s managed data warehouse and analytics platform. Stores very large volumes of data and runs queries over them without you managing servers. If a company has chosen Google Cloud, BigQuery is usually the centre of its analytics.

Microsoft Azure’s analytics and data processing platform, combining storage, processing and analysis. For years it was the centre of Microsoft’s data ecosystem. Microsoft is now pushing Microsoft Fabric, so new Azure projects increasingly mention Fabric instead.

Hearing “Synapse” tells you the cloud, not the productIn practice, when Synapse comes up in a conversation the most useful thing it tells you is that the project lives on Microsoft Azure. What the client actually needs may end up being Fabric, Databricks or something else entirely.

Microsoft’s unified analytics platform, bringing storage, data engineering, warehousing and Power BI reporting into one product. It is where Microsoft is steering new Azure data projects, and it is increasingly the default answer inside Microsoft-heavy organisations.

AWS’s managed data warehouse. Plays the role that BigQuery plays on Google Cloud and that Synapse or Fabric play on Azure: the place where structured data is stored and queried for analytics.

A repository that holds data as it arrives, in whatever shape it arrives: files, logs, exports, images, raw dumps. Cheap and flexible, because nothing has to be modelled before it lands. The cost is that a lake with no governance quietly becomes a place where data goes to be forgotten.

A repository that holds structured, modelled data ready for analysis: sales by customer, monthly billing, margins by region. Data has to be cleaned and shaped before it goes in, which is more work up front and far less work every time somebody asks a question.

An architecture that tries to keep the cheap, flexible storage of a lake and add the structure, reliability and query performance of a warehouse on top of it. It is the pattern behind most modern platform designs, and the one Databricks built its product around.

AWS’s object storage service, and one of the foundations of most data architectures built on AWS: files, datasets, logs, backups, raw data, processing output. Very often it is the data lake.

Google Cloud’s object storage service, usually shortened to GCS. Does essentially the same job as Amazon S3 within the Google Cloud ecosystem.

Microsoft Azure’s object storage. ADLS — Azure Data Lake Storage — is the variant built for analytics workloads. Together they are the Azure equivalent of S3 or GCS.

The set of automated steps that take data from where it is produced to where it is consumed: extract it, clean it, transform it, load it, and do it again tomorrow without anybody pressing a button. When people say a data project failed, they usually mean the pipeline stopped being trustworthy.

Two orderings of the same three steps. ETL — extract, transform, load — cleans the data before storing it. ELT — extract, load, transform — stores it raw first and transforms it inside the destination platform, which is what modern cloud warehouses are fast enough to allow.

The tool that organises, schedules and supervises data processes. Think of a conductor: at 02:00 pull this data, when it finishes run this transformation, then refresh this table, and raise an alert if any step fails. That job is called pipeline orchestration.

Airflow orchestrating PySpark at the edge →

Google Cloud’s managed Airflow service. Composer is Airflow, run and maintained by Google.

Not a different technologyIf a client says “we’re on GCP and we use Composer”, you can read it as “they use Airflow inside Google Cloud”. An engineer who knows Airflow well transfers to Composer with very little friction — which matters when you’re deciding whether a stack mismatch is real or cosmetic.

Microsoft Azure’s service for integrating, moving and orchestrating data between systems — ERP, SQL Server, files, APIs, external applications — and delivering it into the data platform. Usually shortened to ADF, and near-universal in Azure-based companies.

Overlaps with Airflow, but isn’t the sameBoth can coordinate pipelines, so they come up in the same conversations. ADF is stronger on connecting to and moving data between systems; Airflow is stronger on orchestrating arbitrary logic. Plenty of Azure platforms run both, each for what it’s good at.

ADF jobs feeding a quality platform →

A tool for transforming and organising data using mostly SQL. The data has usually already landed in a warehouse, and dbt turns it from raw tables into structured, documented, tested business concepts: sales by customer, monthly billing, margin by region.

Doesn’t replace your warehousedbt is not an alternative to Databricks, Snowflake or BigQuery — it runs on top of them. Snowflake + dbt and BigQuery + dbt are both completely normal combinations.

Batch means processing data in scheduled blocks: every night we process the day’s sales. Streaming means processing events as they happen: we want to see each sale as it occurs. Streaming is more expensive to build and to run, and it is genuinely necessary far less often than it gets asked for.

The best-known technology for carrying large volumes of information continuously and in near real time: transactions, application events, sensor readings, user activity, logs. If Kafka is in the requirements, the project is usually a fairly technical data engineering job.

Kafka moving edge data in real time →

Google Cloud’s managed service for sending and receiving events or messages between systems, used in real-time and event-driven architectures.

Conceptually close to Kafka, not identicalKafka is a streaming platform you can deploy in many ways, anywhere. Pub/Sub is a managed Google Cloud service. The concepts transfer well between them; the operational realities do not.

Microsoft’s business intelligence and data visualisation tool, and by some distance the most common reporting layer in European mid-market companies. Connects to data sources, models the data and produces the dashboards a management team reads.

SAP accounting data surfaced in Power BI →

A key performance indicator: a single number that is supposed to tell you whether something is going well. The hard part is almost never calculating it — it is agreeing on its definition across departments so that finance and sales don’t produce two different revenue figures from the same data.

The layer that defines what each business concept means in the reporting tool: what counts as revenue, what counts as an active customer, how a margin is calculated. It is where a definition lives once instead of being re-invented in every dashboard.

Whether the data is complete, correct, consistent and current enough to be used. Quality problems that are only detected in the reporting layer have already cost you the decision that was made on them, which is why checks belong upstream in the pipeline.

Quality rules and AI-written reports →

Composer is Airflow, managed by Google. Someone fluent in Airflow is productive in Composer almost immediately.

The idea of object storage carries over intact. What has to be learned is the surrounding Google Cloud specifics, not the model itself.

Many concepts overlap and transfer well. They are still different products with different cost models and different operational behaviour, and anyone presenting them as identical is skipping the part that matters.