Add global Lakehouse Schemas for Managing Batch Diagnostic Information
- Dominant language
- JavaScript
- Stars
- 266
- Forks
- 380
- Avg merge
- 5d 8h
- Merged PRs (30d)
- 9
Description
## What are the schemas that are affected by the issue
No existing schemas are impacted.
New schemas created are:
1. aep-lakehouse-batch-input-files-timeseries.schema.json
2. aep-lakehouse-batch-errors-timeseries.schema.json
## What are examples of products that are impacted by the issue
AEP Foundation/Data Landing
Relevant Documentation - https://experienceleague.adobe.com/en/docs/experience-platform/ingestion/quality/error-diagnostics
Internal JIRA - https://jira.corp.adobe.com/browse/PLAT-214882
## Context
We are deprecating the Data Access API service. which is provides access to Batch Diagnostic data. The need for this deprecation has been accelerated due to support in AWS where the Data Access API is unavailable. The Lakehouse team is providing alternative access to this data through system datasets that reference these global schemas.
Contributor guide
Research direction
Start by reviewing the two named schema files, aep-lakehouse-batch-input-files-timeseries.schema.json and aep-lakehouse-batch-errors-timeseries.schema.json, alongside the linked error-diagnostics documentation. Confirm the required batch diagnostic fields and schema conventions before defining both new global schemas; done means both schemas are created without altering existing schemas.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- json
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100