<![CDATA[This Week in Open Lakehouse]]>This Week in Open Lakehouse is a weekly data engineering podcast discussing the latest in open source news, including query engines, data catalogs, open table formats, orchestrators, and streaming technologies. Produced by the Databricks DevRel team. https://openlakehouse.io/

]]>openlakehouse.ioRiverside.fm (https://riverside.com)Fri, 02 Oct 2026 03:19:25 GMTFri, 18 Sep 2026 21:41:51 GMT60Lisa Cao and Scott Haines<p>This Week in Open Lakehouse is a weekly data engineering podcast discussing the latest in open source news, including query engines, data catalogs, open table formats, orchestrators, and streaming technologies. Produced by the Databricks DevRel team. <a href="https://openlakehouse.io/" rel="noopener noreferrer nofollow" target="_blank">https://openlakehouse.io/</a></p>episodicLisa Cao and Scott Hainescarly.akerly@databricks.comno<![CDATA[XML Functions, Polars 2.0, delta-rs, and Agent Thrashing Policies ]]>In this episode, Lisa and Scott explore recent updates in ecosystem technologies including Apache Iceberg, Apache Flink, Polars, delta-rs, highlighting their frustrations in agent workflows and a potential oversight on the importance of XML.

  • Iceberg 1.12: REST catalogs, C++ support, equality deletes, and interoperability
  • Polaris catalog governance, lineage, pagination, and deletion policies
  • Flink updates, including State Fun deprecation and native XML functions
  • Polars 2.0, streaming execution, lazy frames, and memory efficiency
  • Delta Rust releases, partition handling, correctness fixes, and migration
  • Parquet performance, FastLanes benchmarks, and Arrow’s Intel macOS support
  • AI agent orchestration with Omnigent, Kubernetes sandboxes, Spark pipelines, and thrash detection
]]>2b036625-518b-4af8-9c98-374e3ec77aa6Tue, 29 Sep 2026 18:00:30 GMT<p>In this episode, Lisa and Scott explore recent updates in ecosystem technologies including Apache Iceberg, Apache Flink, Polars, delta-rs, highlighting their frustrations in agent workflows and a potential oversight on the importance of XML. </p><ul><li>Iceberg 1.12: REST catalogs, C++ support, equality deletes, and interoperability</li><li>Polaris catalog governance, lineage, pagination, and deletion policies</li><li>Flink updates, including State Fun deprecation and native XML functions</li><li>Polars 2.0, streaming execution, lazy frames, and memory efficiency</li><li>Delta Rust releases, partition handling, correctness fixes, and migration</li><li>Parquet performance, FastLanes benchmarks, and Arrow’s Intel macOS support</li><li>AI agent orchestration with Omnigent, Kubernetes sandboxes, Spark pipelines, and thrash detection</li></ul>no01:06:59XML Functions, Polars 2.0, delta-rs, and Agent Thrashing Policies full<![CDATA[FileTypes, Language-Agnostic UDFs, Parquet Versioning, and Iceberg Updates]]>In this episode, Lisa and Scott explore recent updates in data ecosystem technologies including Apache Iceberg, Apache DataFusion, Apache Parquet, and the Lakekeeper Iceberg Rest Catalog, highlighting their implications for data management and interoperability.

  • Iceberg Data Fusion integration
  • Parquet 2.14 release and versioning
  • Nanosecond timestamp support in Parquet
  • Iceberg 1.12 release and V4 specification
  • Iceberg rest catalog support in Unity Catalog
  • Lakekeeper updates and multimodal support

]]>6b23aea8-db39-43cc-84b5-ac7df5c0594dFri, 18 Sep 2026 23:10:48 GMT<p>In this episode, Lisa and Scott explore recent updates in data ecosystem technologies including Apache Iceberg, Apache DataFusion, Apache Parquet, and the Lakekeeper Iceberg Rest Catalog, highlighting their implications for data management and interoperability.<br /></p><ul><li>Iceberg Data Fusion integration</li><li>Parquet 2.14 release and versioning</li><li>Nanosecond timestamp support in Parquet</li><li>Iceberg 1.12 release and V4 specification</li><li>Iceberg rest catalog support in Unity Catalog</li><li>Lakekeeper updates and multimodal support<br /></li></ul><p></p>no00:23:07FileTypes, Language-Agnostic UDFs, Parquet Versioning, and Iceberg Updatesfull<![CDATA[Nanosecond Timestamp Precision, Iceberg-Datafusion,Spark Connect Rust Client, and Parquet Types]]>The biggest throughline this week is format-layer consolidation. Apache Parquet 2.14.0 shipped as a final release, carrying chronological ordering for INT96 timestamps and Variant documentation fixes, while a parallel dev-list debate over versioning semantics and reader behavior for unsupported versions shows the community is actively thinking about long-term compatibility contracts, not just feature additions.

One layer up, Lance v12 is in a sustained beta sprint, shipping six beta releases in a single week. The work spans file format stabilization (resolving the stable format to 2.2), distributed vector search in Java, in-memory WAL improvements, and IVF index optimization. This is not routine maintenance; it reads as a deliberate push to lock down a production-ready v12 surface before a GA cut.

At the query engine and integration layer, two parallel votes to move iceberg-rust's DataFusion integration into the Apache DataFusion project are the most structurally interesting governance event of the week. If both votes pass, Iceberg catalog access becomes a first-class DataFusion concern rather than a satellite project, which changes how engine builders think about Iceberg adoption. Separately, the Spark Connect Rust client reached RC2 for its 4.2.0 release, and Spark 4.3.0 RC1 is in vote, keeping the Spark release cadence unusually active.

]]>dc867db8-d3f9-468f-b326-99a8305650e9Fri, 18 Sep 2026 22:00:18 GMT<p>The biggest throughline this week is format-layer consolidation. Apache Parquet 2.14.0 shipped as a final release, carrying chronological ordering for INT96 timestamps and Variant documentation fixes, while a parallel dev-list debate over versioning semantics and reader behavior for unsupported versions shows the community is actively thinking about long-term compatibility contracts, not just feature additions.<br /></p><p>One layer up, Lance v12 is in a sustained beta sprint, shipping six beta releases in a single week. The work spans file format stabilization (resolving the stable format to 2.2), distributed vector search in Java, in-memory WAL improvements, and IVF index optimization. This is not routine maintenance; it reads as a deliberate push to lock down a production-ready v12 surface before a GA cut.<br /></p><p>At the query engine and integration layer, two parallel votes to move iceberg-rust's DataFusion integration into the Apache DataFusion project are the most structurally interesting governance event of the week. If both votes pass, Iceberg catalog access becomes a first-class DataFusion concern rather than a satellite project, which changes how engine builders think about Iceberg adoption. Separately, the Spark Connect Rust client reached RC2 for its 4.2.0 release, and Spark 4.3.0 RC1 is in vote, keeping the Spark release cadence unusually active.</p><p></p>no01:16:11Nanosecond Timestamp Precision, Iceberg-Datafusion,Spark Connect Rust Client, and Parquet Typesfull