Databricks on Wednesday introduced a new version of data lake Advances, dubbed Delta Lake 3.0, in order to counter the growing popularity of apache iceberg Tables used by competitors snowflake.
As part of Delta Lake 3.0, the company introduced a new universal table format, dubbed UniForm, that will allow organizations to use Data Lake with other table formats such as Apache Iceberg and apache hoodyThe company said.
A data lake warehouse is a data architecture that provides storage and analysis capabilities, in contrast to the concepts of data lakes, which store data in a native format, and data warehouses, which store structured data (often in SQL appearance).
Databricks said UniForm eliminates the need to manually convert files from different data lakes and data warehouses while performing analytics or building AI models.
The new table format, according to the analysts, is Databricks’ strategy to connect its data lake with the rest of the world and take on competition Snowflake, especially on the back of Apache Iceberg garnering more multi-vendor support in the past few years.
“With UniForm, Databricks are basically saying, if you can’t beat them, join them,” said Tony Baer, principal analyst at dbInsight, likening the battle between table formats to the ones between Apple’s iOS and Google’s Android.
However, Baer believes that adoption of lakehouses will depend on the ecosystem they provide and not just stream formats.
“House-of-lake adoption is still very preliminary because ecosystems have only recently taken shape and most companies are still learning what Home-of-the-Lake is,” Baer said, adding that Houses-of-the-Lake may see meaningful adoption a year from now.
Unlike Baer, Databricks said Delta Lake saw nearly 1 billion downloads a year. Last year, the company Open source delta lick demo This according to the company has seen the lake get updates from contributing engineers from AWS, Adobe, Twilio, eBay and Uber.
Delta core and liquid assembly
As part of Delta Lake 3.0, the company also introduced two other features – Delta Kernel and Liquid Pooling.
According to Databricks, the Delta Kernel handles connector fragmentation by ensuring that all connectors are created using the Delta core library that implements the Delta specification.
The company said this alleviates the need for enterprise users to update Delta connectors with each new release or protocol change.
According to SanjMo Principal Analyst Sanjeev Mohan, the Delta Kernel is like a connector dev kit that strips out many of the essential details and instead provides a set of static elements. APIs.
“This reduces the complexity and time to build and deploy connectors. We expect system integrators will now be able to accelerate the development and deployment of connectors, thereby further expanding the Databricks partner ecosystem,” Mohan said.
liquid gathering It was introduced to address performance issues around data read and write operations, Databricks said.
In contrast to traditional methods such as cell-style partitioning which complicates data management due to its use of static data layout to improve read and write performance, liquid aggregation provides a flexible data layout format that Databricks claims will provide cost-effective aggregations as data volume increases.
Copyright © 2023 IDG Communications, Inc. All Rights Reserved.
ليست هناك تعليقات:
إرسال تعليق