kedro-org / kedro-org/kedro-devrel
Conduct market research on c̶a̶t̶a̶l̶o̶g̶s̶ metastores
- Dominant language
- No language data
- Stars
- 0
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
- June 3rd: Snowflake announces Polaris Catalog, an open source catalog for Apache Iceberg "in the next 90 days" ([announcement](https://www.snowflake.com/blog/introducing-polaris-catalog/))
- June 4th: Databricks announces it has agreed to acquire Tabular ([announcement](https://www.databricks.com/company/newsroom/press-releases/databricks-agrees-acquire-tabular-company-founded-original-creators))
- June 13th: Databricks announces the open sourcing of Unity Catalog https://github.com/unitycatalog/unitycatalog ([announcement](https://www.databricks.com/company/newsroom/press-releases/databricks-open-sources-unity-catalog-creating-industrys-only-open))
At the moment, each platform offers only full read & write capabilities to their own catalog, and read-only capabilities for competitors:

([source](https://seattledataguy.substack.com/p/apache-iceberg-what-is-it))
And what's more important: data catalogs aren't new, but we're seeing catalogs created for different use cases and business needs: technical, business, and operational ([source](https://royondata.substack.com/p/the-race-to-own-open-data)).
These are just some open source ones[^1] that have been in the news recently. But there's also [Apache Nessie](https://projectnessie.org/), the Hive Metastore, the Iceberg REST Catalog, probably others I'm missing. Then there are the commercial, vendor-driven ones.
[^1]: counting Polaris as open source
And then we have... the Kedro Catalog!
We've sometimes got questions on "how does the Kedro Catalog compare to the Unity Catalog" - and the answer is that they're complementary, but this is not immediately clear to users (see https://github.com/kedro-org/kedro-plugins/issues/542).
It's very clear that this is going to be a hot topic of discussion in the data engineering space in the coming months so we should have a good answer to how does Kedro interact with all these.
Contributor guide
No contributing guide indexed for this repository
Research direction
Review the linked announcements and catalog sources for Polaris Catalog, Tabular, Unity Catalog, Apache Nessie, Hive Metastore, and the Iceberg REST Catalog, along with the related Kedro Catalog discussion in kedro-plugins issue 542. Done means establishing a clear answer to how Kedro interacts with these catalogs and how its catalog compares with Unity Catalog and other use cases.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering, documentation
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100