Migrate exsting data (metadata) to the new dataset system

Description of the issue

Since the latest release, v1.9.0 we introduced the dataset concept finally also on the production OEP. This will help us a lot to maintain and communicate data available on the OEP. This again puts metadata first so all table resources and all datasets have their own management system. Tables can be assigned to multiple datasets this way.

We have ~ ~5k tables available on the OEP. Some of them are already grouped into datasets using the oemetadata v2.0.4 spec. The new dataset system does not register the existing data automatically, so we need a migration pipeline to ingest.

Ideas of solution

A processing script and database migration to ingest all existing data resources (their metadata json objects). It then attempts to find grouping parameters and collects them as a report. Dev will pick this observation up and create the actual grouping.

We should still inform the users that we migrated their data. We can't always be fully sure if the patterns match the correct resources for a dataset or if some tables actually are intended to be added to any dataset.

On the API end, we should review the single resource rule for table object when storing the metadata

is it covered everywhere? Will all API operations catch the correct resource even if multiple are sent?
Forward to the dataset api if a oemetadata dataset with multiple tables was sent in the request. Or provide a response with feedback that we added the oemetadata to the table but filtered out only the single table. Then also respond to the hint: We offer the dataset API; there you can create all tables that are part of an OEMetadata dataset in one go - if they are properly described in the oemetadata (=/> version 2.0.4)

Workflow checklist

• I am aware of the workflow in
  CONTRIBUTING.md