Data engineering · in detail
Process, deliverables, pricing and what you’ll actually build.
Somebody has to write the pipelines that move data from source systems to dashboards. This is the work that turns architecture diagrams into running code—tested, monitored, and maintained for the next team.
Process step-by-step
Engineering work follows a clear build sequence.
You can’t transform data before you’ve ingested it. You can’t optimize queries before you know which ones run slow. Every pipeline starts the same way.
Source connection & schema discovery
We connect to your source systems—SAP, Salesforce, PostgreSQL, APIs—and extract their schemas. We identify primary keys, foreign keys, update patterns, and data types. We test extraction logic on a subset of tables to validate connectivity, permissions, and performance. This isn’t full ingestion. It’s proof that the pipeline can run.
Deliverable: Connection documentation and sample data extracts from each source.
Bronze layer build & incremental loading
We build ingestion pipelines that land raw data in your data lake or warehouse. We implement incremental loading where possible (using change data capture, watermarks, or timestamps) to avoid full-table scans. We add logging, error handling, and retry logic. Every table gets its own pipeline. Every pipeline runs on a schedule.
Deliverable: Automated Bronze pipelines with monitoring and alerting.
Silver/Gold transformation & data quality
We transform raw data into clean, joined, aggregated tables. We apply business logic—currency conversions, date calculations, category mappings. We add data quality checks (null rates, duplicate detection, schema drift alerts). We version every transformation so changes are traceable. This is the layer analysts and dashboards query.
Deliverable: Transformation pipelines with data quality tests and lineage tracking.
Testing, deployment & handoff
We write unit tests for transformation logic and integration tests for end-to-end data flow. We deploy pipelines through CI/CD (dev → test → prod) with approval gates. We document dependencies, schedules, and failure scenarios. At the end, we run a technical handoff session with your team to transfer ownership.
Deliverable: CI/CD setup, test suite, runbooks, and handoff documentation.
Detailed deliverables
What you actually receive
These are production pipelines, not proof-of-concept scripts. They run daily, handle failures, and log every operation. Your team can maintain them without us.
Bronze ingestion pipelines
Automated extract jobs that pull data from source systems and land it in your data lake or warehouse. Each pipeline includes incremental loading logic (where supported), error handling, retry policies, and logging. Pipelines are parameterized so you can re-run specific date ranges or tables without changing code.
Transformation DAGs (Silver/Gold)
SQL or Python-based transformation logic organized into directed acyclic graphs (DAGs). Each transformation declares its dependencies—which tables it reads, which it writes, in what order. This makes it possible to backfill data, debug failures, and understand lineage without reading all the code.
Data quality tests
Automated checks that run after every pipeline execution: null rate thresholds, uniqueness constraints, referential integrity, schema validation, row count deltas. Failures trigger alerts (email, Slack, PagerDuty) with enough context to diagnose the issue without SSH-ing into a server.
CI/CD deployment setup
GitHub Actions, Azure DevOps, or GitLab CI pipelines that test and deploy code across environments (dev, test, prod). Changes go through pull requests, automated tests, and approval gates. Rollbacks are one command. This prevents “works on my machine” production incidents.
Runbooks & troubleshooting guides
Step-by-step instructions for common failure scenarios: source system down, schema change, row count spike, query timeout. Each runbook includes where to check logs, which metrics to inspect, and how to escalate. This lets junior engineers handle incidents without escalating to seniors.
Monitoring dashboards
Grafana, Databricks, or platform-native dashboards showing pipeline execution times, row counts processed, data freshness lag, error rates, and cost per table. You can see at a glance which pipelines are slow, which failed overnight, and whether data is up-to-date.
Related case
Education: Marketing attribution & ROI modeling
An education client needed to unify data from Facebook Ads, Google Ads, Google Analytics, TikTok Ads, Spotify Ads, SEMRush, and Google Search Console with financial data to calculate true marketing ROI and model conversion funnels across landing pages. We built incremental ingestion pipelines for 7 marketing sources, transformation logic that matched campaign spend to revenue, and KPI tables that tracked cost-per-acquisition and lifetime value by channel— letting them reallocate €180k/year from low-ROI platforms to high-converting campaigns.
Read the full case →Pricing & what affects it
Transparent pricing context
Base engineering project
€40/hour
Estimated 160–240 hours · 4–6 weeks
What makes the price go up
- Number of tables/endpoints: Ingesting 100 tables takes longer than 10. Each requires extraction logic, schema mapping, and incremental loading patterns.
- Complex transformations: Simple joins and aggregations are fast. Multi-step logic with business rules, date arithmetic, and currency conversions require more testing.
- Real-time or near-real-time requirements: Batch pipelines (daily, hourly) are simpler than streaming (CDC, message queues, event-driven triggers).
- Legacy source systems: Extracting from old on-prem databases with no APIs, undocumented schemas, or restricted network access requires custom connectors and VPN setup.
What keeps the price down
- Standard connectors available: If your sources have Fivetran, Airbyte, or native cloud connectors, we use those instead of writing custom extraction code.
- Simple aggregations: Sum, count, average, group by—if that’s all you need, transformation logic is straightforward.
- Batch-only workloads: Daily overnight pipelines are simpler and cheaper than hourly or real-time ingestion.
- Existing architecture in place: If you already have Bronze/Silver/Gold layers defined, we’re implementing, not designing from scratch.
Team reinforcement
€37.5/hour
Embedding an engineer in your team for 6+ months
Pipeline maintenance
€40/hour
Ongoing monitoring, bug fixes, and schema change handling
Questions
FAQ: Data engineering
Do I need an engineer if I already have Fivetran or Airbyte?
Not for basic ingestion. If your sources have pre-built connectors and you only need raw data in your warehouse, tools like Fivetran handle it. Engineering becomes necessary when you need custom transformations, business logic, data quality checks, or orchestration across multiple systems that connectors don’t cover.
What’s the difference between ELT and ETL?
ETL (Extract, Transform, Load) transforms data before loading it into the warehouse—common with older on-prem systems where storage was expensive. ELT (Extract, Load, Transform) loads raw data first, then transforms it in the warehouse using SQL. Modern cloud platforms (Snowflake, Databricks, BigQuery) are optimized for ELT because compute is cheaper than moving data around.
How do you handle schema changes in source systems?
We use schema evolution patterns: adding new columns doesn’t break existing pipelines, removing columns triggers alerts, renaming columns is handled with mapping layers. We version transformations so changes are traceable. For critical tables, we add schema validation tests that fail if unexpected changes appear, forcing a review before data propagates downstream.
Next steps
Engineering work comes after architecture.
If you know where your data needs to go and what it needs to look like, this is where you build the pipelines that get it there.