6 Data Lakehouse
6.1 Introduction to the Lakehouse Architecture
As discussed in the previous chapter, both the Data Warehouse (consistent but rigid and expensive) and the Data Lake (flexible but prone to disorganization and lacking transactional support) pose major challenges for modern data teams. To combine the SQL performance and governance of a Data Warehouse with the low cost and scalability of a Data Lake, the Data Lakehouse architecture emerged.
This characterization describes historical trends rather than absolute rules: elastic Data Warehouses and well-governed Data Lakes exist. The challenge arises when an organization maintains both as disconnected platforms. Data is copied from the lake to the warehouse for reporting, then prepared again for data science. Every copy adds latency, cost, and the risk of different teams operating on conflicting versions of truth.
The Lakehouse represents a natural evolution of the Data Lake. It retains object storage, multi-format ingestion, and the decoupling of storage from compute, while incorporating a table, metadata, and governance layer capable of enforcing ACID transactions, schema controls, access policies, and reliable SQL performance. It does not mean placing a Data Warehouse next to a Data Lake, but turning lake files into managed tables without relinquishing openness and flexibility.
6.2 The Best of Both Traditions
The term Lakehouse signifies convergence. From the Data Lake, it inherits scale, low-cost object storage, multi-format compatibility, and multi-engine accessibility. From the Data Warehouse, it restores necessary enterprise-grade guarantees: verifiable quality, understandable domain models, concurrent queries, and centralized governance.
This unification does not mean all data must undergo identical transformations. A newly ingested log file can coexist with an executive-certified reporting table, provided the architecture distinguishes quality tiers, enforces access control, and tracks lineage.
To understand how this technology operates in practice, we examine the five logical layers composing a robust Data Lakehouse architecture.
6.3 The Five Layers of Data Lakehouse Architecture
The following diagram maps out the components of a Lakehouse architecture, from source systems to downstream analytics:
SQL databases, ERPs
JSON, logs, IoT events
Audio, images, PDFs
Open columnar files on disk (Parquet, ORC, etc.) hosted in cost-effective buckets.
ACID transactional control, schema enforcement, data lineage, and time travel versioning.
6.4 Detailed Overview of the Five Layers
The technical responsibilities of each logical layer in this architecture are detailed below:
6.4.1 1. Ingestion Layer
The entry point connecting diverse data sources (structured, semi-structured, or unstructured) with the platform. It supports two core operating modes: * Batch Processing: Scheduled bulk ingestion of accumulated datasets (e.g., nightly transaction dumps from PostgreSQL). * Streaming Processing: Continuous ingestion of event streams as they occur (e.g., web clickstreams or IoT sensor telemetry).
6.4.2 2. Storage Layer
The Data Lakehouse implements physical storage over a Data Lake built on cost-effective, scalable object storage systems (such as Amazon S3, Google Cloud Storage, or Azure ADLS). * Open Formats: Unlike traditional DWH engines (which encapsulate data in proprietary binary formats), the Lakehouse persists data using open-source, highly efficient columnar file specifications (such as Apache Parquet or ORC). This prevents vendor lock-in.
6.4.3 3. Metadata Layer
The core innovation of the Data Lakehouse. A logical metadata layer sits over raw physical files in object storage, bridging the gap with relational database features. It delivers: * ACID Transactions: Ensures concurrent processes can read and write Parquet files simultaneously without creating data conflicts or corrupt files. * Governance and Compliance: Enforces strict schema validations (Schema Enforcement), preventing data rot or ungoverned schema changes. * Time Travel: Maintains snapshot version histories of metadata, allowing queries against historical states at specific points in time.
6.4.4 4. API Layer
Provides standardized interfaces for analytical engines to query storage natively without data movement or copying: * SQL APIs: Enables direct querying via standard SQL syntax using optimized query engines (like Spark SQL or Photon). * Declarative Dataframe APIs: Supplies programmatic interfaces (like PySpark or Polars Dataframes) for complex big data transformations. * Metadata APIs: Exposes data lineage and snapshot histories for auditing and governance tools.
6.4.5 5. Consumption Layer
The presentation tier where business value is extracted. Unlike traditional architectures requiring separate database engines, the Lakehouse directly serves: * Business Intelligence (BI): Interactive reporting and live dashboards (via Metabase, Tableau, Power BI) querying tables with DWH-level performance. * Data Science and Machine Learning: High-throughput raw data access for AI model training (via Python, R, PyTorch) directly against columnar storage.
These five layers represent functional responsibilities rather than mandated separate software products. Managed cloud platforms may combine several layers, whereas open-source architectures distribute them across tools. The essential requirement is establishing clear contracts between layers: storage location, table management, execution engines, and policy enforcement points.
6.5 The Key Innovation: From Files to Reliable Tables
A traditional Data Lake can store thousands of well-organized Parquet files while still lacking transactional behavior. If two processes write concurrently, readers might observe partial updates. Furthermore, Parquet alone does not track which specific files comprise a valid table snapshot at a given timestamp.
Modern table formats—such as Apache Iceberg, Delta Lake, and Apache Hudi—add a metadata protocol over raw files. These protocols track versions, schemas, partitions, and valid transactions. As a result, processing engines treat a directory not as a loose file collection, but as a fully versioned table with identity and history.
This distinction resolves a common point of confusion: Parquet is a file format; Iceberg, Delta Lake, and Hudi are table formats. Parquet organizes columnar bytes inside a single file; table formats coordinate file manifests and transaction logs across time.
6.6 Medallion Architecture: Bronze, Silver, and Gold
A standard design pattern for managing data quality within a Lakehouse is the Medallion Architecture. Bronze, Silver, and Gold represent progressive quality states rather than physical locations. This pattern preserves raw data for lineage while publishing refined datasets for business decision-making.
The Bronze layer is an audit landing zone; Silver produces reusable enterprise entities; Gold materializes business metrics. Organizations frequently host multiple Gold tables, each powering specific analytical products and decision thresholds.
6.7 Decoupling Storage and Compute
Cloud platforms made another core property viable: storing data once while allocating compute dynamically based on workload demands. Executive dashboards, Spark data transformations, and ML training require different compute profiles. In a Lakehouse, these execution engines scale independently while querying shared, governed storage.
Decoupling provides elasticity, though query performance still relies on storage design. Table formats, catalog metadata, file sizing, partitioning, and caching strategies remain vital to performance. The Lakehouse promise is specialized execution engines working over shared data contracts.
7 Lakehouse in Modern Cloud Platforms
Organizations rarely assemble every layer from scratch; they combine managed cloud services for storage, cataloging, processing, governance, and reporting. Major cloud providers offer distinct implementations of this architecture:
Tables & Governance: BigLake and catalog
Engines: BigQuery and Spark
Consumption: SQL, BI, Data & AI
Catalog & Governance: Glue and Lake Formation
Engines: Athena, Redshift, EMR, Glue
Consumption: SQL, BI, Data & AI
Tables: Delta
Engines: Fabric, SQL, Spark
Consumption: Power BI, Data & AI
7.1 Google Cloud: BigQuery, BigLake, and Open Storage
In Google Cloud, BigQuery provides a serverless SQL engine, while BigLake extends table governance and access controls over Cloud Storage, Amazon S3, or Azure Blob Storage. Apache Iceberg managed tables allow data to reside in open storage while remaining queryable across engines. See the official BigLake overview and BigQuery Apache Iceberg tables.
BigQuery can also function as a traditional Data Warehouse. The Lakehouse paradigm emerges when SQL compute, object storage, open formats, and unified governance query shared datasets without requiring pre-loading into proprietary native tables.
7.2 Amazon Web Services: S3, Catalogs, Governance, and Multi-Engine Access
In AWS, Amazon S3 serves as primary storage, AWS Glue Data Catalog maintains table definitions, and AWS Lake Formation centralizes security permissions. Engines including Amazon Athena, Redshift Spectrum, EMR, and Glue query and transform governed data. AWS Lakehouse patterns integrate S3 and Redshift storage via open catalogs, exposing Apache Iceberg tables to compatible engines. See AWS Lakehouse Architecture and AWS Lake Formation Documentation.
This ecosystem illustrates a key Lakehouse trait: the architecture is a coordinated suite of specialized services rather than a single monolith.
7.3 Microsoft Azure: OneLake, Fabric, and Azure Databricks
Microsoft supports multiple implementations. In Microsoft Fabric, OneLake provides unified storage over ADLS Gen2; Lakehouse items manage files and tables, while Spark, SQL endpoints, and Power BI query underlying data natively. Fabric uses Delta Lake as its open table format to eliminate data duplication across analytics tools. See the Microsoft Fabric Overview and Fabric Data Warehouse Architecture.
Alternatively, Azure Databricks pairs Delta Lake for transactions with Unity Catalog for governance, Spark for processing, and Databricks SQL for BI. Microsoft defines this pattern as combining the strengths of Data Lakes and Data Warehouses. See What is a Data Lakehouse? in Azure Databricks.
7.4 Evaluating Cloud Provider Patterns
Service equivalences vary across vendors. BigQuery, Redshift, Athena, Fabric Warehouse, and Databricks SQL feature different execution and cost models. A clear evaluation begins with requirements:
- Where will physical data reside, and who controls access policies?
- Which open table format ensures cross-engine interoperability?
- Which catalog serves as the single source of truth for schemas and lineage?
- Which execution engines handle SQL, streaming, ETL, and ML workloads?
- How will cost management, observability, and disaster recovery be handled?
8 Practical Example: E-Commerce Platform
Consider an e-commerce company storing order transactions in a relational database, clickstreams in JSON logs, product media in object storage, and customer support chats in text logs. Finance requires certified sales figures; Marketing needs real-time session metrics; Data Science trains recommendation algorithms.
In a traditional architecture, each team builds separate ETL pipelines and database copies. In a Lakehouse, all raw events land in Bronze; customer, product, and session entities are validated in Silver; sales, conversion, and churn KPIs are published in Gold. BI tools query Gold tables, while ML models train on Silver entities combined with authorized unstructured assets—all governed under a single catalog.
9 Benefits, Trade-offs, and Architectural Decisions
While a Lakehouse reduces data duplication and simplifies infrastructure, it requires rigorous data engineering. Achieving value demands clear data contracts, automated testing, governance policies, file compaction, partitioning strategies, and cost controls.
Key benefits include:
- A unified data platform for BI, data science, streaming, and AI.
- Reduced data movement between isolated storage engines.
- Independent scaling of storage and compute resources.
- Open file and table formats preventing vendor lock-in.
- Historical versioning, auditing, and time travel via table metadata.
Challenges to manage:
- True engine interoperability depends on feature support across table format implementations.
- Small-file problems, poor partitioning, or missing column stats degrade SQL queries.
- Catalogs without designated owners or policies fail to enforce governance automatically.
- Elastic compute scaling can increase costs without quotas and budget monitoring.
- Poorly managed Gold layers risk recreating the data silos the Lakehouse was designed to replace.
10 Conclusion
The Data Lakehouse represents a modern data platform architecture: taking scalable object storage as a flexible foundation and adding the transactional guarantees and governance that made the Data Warehouse valuable. Its goal is not eliminating data warehouses, but removing artificial boundaries between storage, analytics, and artificial intelligence.
Google Cloud, AWS, and Microsoft implement this concept through distinct combinations of object storage, open table formats, metadata catalogs, and query engines. Platforms evolve, but core principles endure. A well-designed Lakehouse persists multi-format data, manages files as reliable tables, supports specialized processing engines, and delivers trusted data products to the business.
The core architectural question is not “which vendor tool should we purchase?”, but “how can our platform deliver open, governed, consistent data across teams without proliferating redundant copies or conflicting definitions?”.