Delta Lake vs Apache Iceberg in 2026: Lakehouse Architecture and Interview Questions

Delta Lake and Apache Iceberg represent the two dominant open table formats powering modern data lakehouses in 2026. Both solve the same fundamental problem—bringing ACID transactions, schema evolution, and time travel to cloud object storage—but take different architectural approaches that matter for production workloads.

Delta Lake vs Iceberg: Core Architecture Differences

Both formats store data as Parquet files on object storage, but their metadata layers differ significantly.

Delta Lake uses a transaction log (_delta_log/) containing JSON files that record every change. Each commit creates a new JSON file, and periodic checkpoints consolidate these into Parquet for faster reads. The Delta Lake protocol specification defines versioned reader/writer features for forward compatibility.

Apache Iceberg maintains a hierarchy of metadata files: a metadata JSON file pointing to manifest lists, which point to manifests containing file-level statistics. This design enables predicate pushdown at the planning stage—the query engine skips entire files before reading any data.

| Feature | Delta Lake | Apache Iceberg | |---------|------------|----------------| | Transaction Log | JSON + Parquet checkpoints | Metadata JSON + manifest files | | Partition Evolution | Requires rewrite | In-place without rewrite | | Engine Support | Spark, Trino, Flink | Spark, Trino, Flink, Dremio, Athena | | Time Travel | Version-based | Snapshot-based with branch/tag | | Schema Evolution | Add/rename columns | Add/rename/reorder/widen columns |

Hidden Partitions: Iceberg's Key Advantage

Traditional Hive-style partitioning exposes partition columns in queries (WHERE year=2026 AND month=7), coupling physical layout to SQL syntax. Iceberg's hidden partitions decouple these concerns.

# iceberg_partition_example.py
from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import StringType, TimestampType, LongType, NestedField
from pyiceberg.partitioning import PartitionSpec, PartitionField
from pyiceberg.transforms import MonthTransform

# Define schema
schema = Schema(
    NestedField(1, "event_id", LongType(), required=True),
    NestedField(2, "event_time", TimestampType(), required=True),
    NestedField(3, "user_id", StringType(), required=True),
    NestedField(4, "event_type", StringType(), required=False),
)

# Hidden partition on month extracted from event_time
partition_spec = PartitionSpec(
    PartitionField(
        source_id=2,  # event_time field
        field_id=1000,
        transform=MonthTransform(),
        name="event_month"
    )
)

catalog = load_catalog("glue", **{"type": "glue"})
catalog.create_table(
    identifier="analytics.events",
    schema=schema,
    partition_spec=partition_spec
)

Queries filter on event_time directly—the engine automatically applies partition pruning without exposing the partition column in the WHERE clause. Partition evolution lets the table switch from monthly to daily partitioning without rewriting existing data.

Delta Lake ACID Transactions and Optimistic Concurrency

Delta Lake implements serializable isolation using optimistic concurrency control. Writers check the transaction log before committing to detect conflicts.

# delta_concurrent_writes.py
from delta import DeltaTable
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
    .config("spark.sql.catalog.spark_catalog", 
            "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
    .getOrCreate()

# Concurrent MERGE operation with automatic conflict resolution
delta_table = DeltaTable.forPath(spark, "s3://bucket/events")

# Merge incoming batch with existing data
delta_table.alias("target").merge(
    source=incoming_df.alias("source"),
    condition="target.event_id = source.event_id"
).whenMatchedUpdateAll() \
 .whenNotMatchedInsertAll() \
 .execute()

# Conflict detection happens automatically during commit
# Failed transactions retry with updated snapshot

Delta Lake's conflict resolution rules allow concurrent appends to succeed while serializing conflicting updates to the same partition.

Time Travel and Data Versioning Comparison

Both formats support time travel, but with different semantics.

-- Delta Lake: Query by version number
SELECT * FROM events VERSION AS OF 42;

-- Delta Lake: Query by timestamp
SELECT * FROM events TIMESTAMP AS OF '2026-07-15 10:00:00';

-- Iceberg: Query by snapshot ID
SELECT * FROM analytics.events FOR SYSTEM_VERSION AS OF 8823749234;

-- Iceberg: Query by timestamp
SELECT * FROM analytics.events FOR SYSTEM_TIME AS OF TIMESTAMP '2026-07-15 10:00:00';

Iceberg 1.5+ adds branching and tagging—create named branches for isolated experimentation, then merge or discard. Delta Lake achieves similar workflows through shallow clones.

-- Iceberg: Create a branch for testing
ALTER TABLE analytics.events CREATE BRANCH experiment;

-- Write to branch without affecting main
INSERT INTO analytics.events.branch_experiment 
SELECT * FROM staging.events WHERE event_type = 'test';

-- Merge branch to main
CALL system.fast_forward('analytics.events', 'main', 'experiment');

Query Engine Compatibility Matrix

Engine compatibility drives format selection for many organizations.

| Engine | Delta Lake Support | Iceberg Support | |--------|-------------------|------------------| | Apache Spark | Native (Databricks, OSS) | Native | | Trino/Presto | Connector | Native | | Apache Flink | Connector | Native (1.16+) | | Dremio | Read-only | Native | | AWS Athena | Limited | Native | | Snowflake | External tables | Native (Iceberg tables) | | BigQuery | External tables | BigLake Iceberg |

For Spark-only environments, Delta Lake's tighter integration offers better performance. Multi-engine architectures benefit from Iceberg's broader compatibility—the Iceberg REST Catalog provides a vendor-neutral API for catalog operations.

Performance Optimization Techniques

Both formats require maintenance operations for optimal query performance.

# delta_optimize.py
from delta import DeltaTable

delta_table = DeltaTable.forPath(spark, "s3://bucket/events")

# Compact small files (bin-packing)
delta_table.optimize().executeCompaction()

# Z-order for multi-column predicates
delta_table.optimize().executeZOrderBy("user_id", "event_time")

# Remove old versions (vacuum)
delta_table.vacuum(retentionHours=168)  # Keep 7 days

-- Iceberg: Rewrite data files for compaction
CALL catalog.system.rewrite_data_files(
  table => 'analytics.events',
  strategy => 'binpack',
  options => map('target-file-size-bytes', '134217728')
);

-- Iceberg: Expire old snapshots
CALL catalog.system.expire_snapshots(
  table => 'analytics.events',
  older_than => TIMESTAMP '2026-07-01 00:00:00',
  retain_last => 10
);

Z-ordering clusters related data together, improving predicate pushdown effectiveness. Both formats benefit from file compaction when ingesting many small files from streaming sources.

Common Data Lakehouse Interview Questions

Prepare for these frequently asked questions in data engineering interviews.

Q: When would you choose Iceberg over Delta Lake?

Iceberg fits better when: (1) multiple query engines access the same tables (Spark, Trino, Flink), (2) partition schemes need to evolve without rewriting data, (3) the organization prefers vendor-neutral open standards. Delta Lake excels in Databricks-centric architectures or pure Spark environments where Unity Catalog provides governance.

Q: How does Iceberg achieve partition evolution without data rewrite?

Iceberg stores partition specs as metadata. When the spec changes, new data uses the new partitioning while existing data retains its original layout. The query planner reads all partition specs and applies the correct pruning logic for each file group.

Q: Explain Delta Lake's checkpoint mechanism.

Delta Lake creates checkpoint files every 10 commits by default. A checkpoint consolidates the transaction log into a single Parquet file for faster reads. The _last_checkpoint file points to the latest checkpoint, and readers reconstruct state by reading the checkpoint plus any subsequent JSON commits.

Q: What is copy-on-write vs merge-on-read?

Copy-on-write (COW) rewrites entire files on updates—higher write cost but faster reads. Merge-on-read (MOR) writes deltas separately, merging at query time—lower write latency but read overhead. Iceberg supports both; Delta Lake primarily uses COW with deletion vectors for soft deletes in recent versions.

Practice these patterns on SharpSkill's data engineering interview questions to solidify understanding before interviews.

Migration Path: Converting Between Formats

Organizations sometimes need to migrate between formats. Both support conversion utilities.

# Convert Delta to Iceberg using Spark
spark.sql("""
    CALL catalog.system.snapshot(
        source_table => 'delta.`s3://bucket/delta_events`',
        table => 'analytics.events_iceberg'
    )
""")

# Convert Iceberg to Delta
from delta.tables import DeltaTable

iceberg_df = spark.read.format("iceberg").load("catalog.analytics.events")
iceberg_df.write.format("delta").save("s3://bucket/delta_events")

Full migration requires careful validation of schema types, partitioning semantics, and downstream consumer compatibility.

Conclusion

• Choose Delta Lake for Spark-native workloads, Databricks environments, and when Unity Catalog governance fits organizational needs
• Choose Iceberg for multi-engine architectures, partition evolution requirements, and cloud-agnostic deployments
• Both formats deliver ACID transactions, time travel, and schema evolution—the right choice depends on existing infrastructure and query engine diversity
• Interview preparation should cover transaction log internals, partition pruning optimization, and COW vs MOR tradeoffs
• File compaction and snapshot expiration remain critical maintenance tasks regardless of format choice