The problem with single-use data platforms
Most enterprise data platforms start as a single-use solution: a specific team has a reporting requirement, an analyst builds a set of pipelines to meet it, and those pipelines run until they don't. The problem appears when a second team wants to use the data. They discover that the pipelines are brittle, the transformation logic is embedded in undocumented SQL, the data quality is variable, and the semantics of key fields differ from what they expect.
This is the consequence of building for one use case rather than for a platform. A platform is a set of shared data assets and infrastructure that multiple teams can build on without needing to understand the details of the data source. Getting from single-use pipelines to a genuine platform requires deliberate architectural work.
The medallion architecture
The medallion architecture — Bronze, Silver, Gold — has become the de facto standard for data lake design, and for good reason: it provides a clear separation between raw data, cleaned and conformed data, and business-level aggregations that maps well to how different teams consume data.
Bronze is the raw ingestion layer. Data lands here in its source format, with minimal transformation. The goal is fidelity: Bronze should be a complete, immutable record of what came from the source. Nothing is discarded, and nothing is modified. If a source system sends malformed records, those malformed records exist in Bronze — you can fix them downstream without losing the original.
Silver is the cleaned, conformed layer. Data here has been validated, deduplicated, and standardised — field names are consistent, data types are correct, null handling is explicit. Silver is where business rules about data quality are enforced. A customer record in Silver should be a clean, reliable representation of a customer across all source systems.
Gold is the business-level layer. Data here is aggregated, joined, and structured for the way specific use cases consume it — dimensional models for analytics, feature tables for ML, denormalised views for reporting. Gold is what analysts and data scientists work with most of the time.
The separation matters because it gives you a clear answer to the question "where do I fix this?" — data quality issues are fixed in Silver, schema changes are absorbed in the Bronze-to-Silver transformation, and business logic changes happen in Gold without touching the underlying data.
dbt as the transformation layer
dbt (data build tool) has become the dominant tool for the Silver and Gold transformation layers in the data platforms we build, and for reasons that go beyond popularity. The critical advantage of dbt is that it treats SQL transformation as software — version-controlled, testable, documented, and reviewable. This addresses the single biggest quality problem with traditional data pipelines: undocumented SQL logic that nobody fully understands.
dbt models are SQL files with dependency management. When you build a Silver model that depends on a Bronze table, dbt tracks that dependency and ensures the execution order is correct. When you add a test to a model — a not-null constraint, a uniqueness test, a referential integrity check — dbt runs that test before downstream models execute, surfacing data quality issues before they propagate.
The documentation generation is also genuinely useful: dbt produces a data catalogue as a side effect of building, meaning the documentation is always current because it's derived from the same code that generates the data.
The semantic layer
The semantic layer sits between the data platform and the tools that consume it — BI tools, analytics notebooks, internal applications. Its purpose is to define business metrics and dimensions in a single place, so that "revenue" means the same thing regardless of which tool is querying it.
Without a semantic layer, every analyst and every BI report defines metrics independently. Revenue might be calculated differently in the finance team's Power BI report and the operations team's Tableau dashboard. When the numbers don't match, nobody knows which one is right. The semantic layer eliminates this by making the definition canonical.
In practice, the semantic layer can be implemented in several ways: a dedicated semantic layer tool (Looker, AtScale, Cube), a dimensional model in the Gold layer with strict naming conventions, or a combination. The right choice depends on the complexity of the metric definitions and the tools consuming them.
Data contracts between producers and consumers
One of the most common failure modes in data platforms is upstream source system changes breaking downstream pipelines. A source table has a column renamed, a null constraint relaxed, or a new status code added — and three dashboards break. The teams involved don't have a shared understanding of what changes are breaking versus non-breaking.
Data contracts formalize the interface between data producers (source systems, pipelines) and consumers (downstream models, applications). A producer agrees to maintain the contract — field names, types, semantics — and consumers can build against it with confidence. Schematised formats like Avro and Parquet enforce contracts at the storage layer. Schema registry tools (Confluent Schema Registry for Kafka-based pipelines) enforce them at the streaming layer.
Implementing data contracts requires cultural work as much as technical work — the teams maintaining source systems need to understand the downstream impact of their changes. The technical tooling makes the impact visible; the organisational agreement makes people act on it.
Data quality as a first-class concern
Data quality monitoring deserves the same engineering attention as application monitoring, but rarely gets it. The typical situation is: pipelines run, data lands, reports are produced, and nobody knows whether the data is correct until an analyst or business user notices a number that doesn't look right.
Automated data quality tests at each layer of the medallion architecture — completeness checks, value range validation, referential integrity, cross-source reconciliation — provide early warning when data quality degrades. dbt's test framework covers the common cases at the transformation layer. Tools like Great Expectations or Monte Carlo cover anomaly detection and drift monitoring at the data layer.
The useful question to ask at the start of a data platform build is: what would a data quality incident look like, and how quickly would we know? If the answer is "we wouldn't know until a business user complained," that's the gap the quality monitoring needs to close.
Related resources
Our healthcare data platform case study documents a FHIR-compliant data lake build on Azure for a 14-facility health network. Our data and analytics service covers how we approach platform builds in more detail.