adobe / adobe/xdm

Add global Lakehouse Schemas for Managing Batch Diagnostic Information

Open
#1,918 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
266
Forks
380
Avg merge
5d 8h
Merged PRs (30d)
9

Description

## What are the schemas that are affected by the issue

No existing schemas are impacted.
New schemas created are:

1. aep-lakehouse-batch-input-files-timeseries.schema.json
2. aep-lakehouse-batch-errors-timeseries.schema.json

## What are examples of products that are impacted by the issue

AEP Foundation/Data Landing
Relevant Documentation - https://experienceleague.adobe.com/en/docs/experience-platform/ingestion/quality/error-diagnostics
Internal JIRA - https://jira.corp.adobe.com/browse/PLAT-214882

## Context
We are deprecating the Data Access API service. which is provides access to Batch Diagnostic data. The need for this deprecation has been accelerated due to support in AWS where the Data Access API is unavailable. The Lakehouse team is providing alternative access to this data through system datasets that reference these global schemas.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the two named schema files, aep-lakehouse-batch-input-files-timeseries.schema.json and aep-lakehouse-batch-errors-timeseries.schema.json, alongside the linked error-diagnostics documentation. Confirm the required batch diagnostic fields and schema conventions before defining both new global schemas; done means both schemas are created without altering existing schemas.

Written by the indexing model from the issue text.

Assessment

Tech stack
json
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.