bcgov / bcgov/Mines-Data-Platform

Production Ingestion Pipelines for Fabric to load on premise data

Open
#339 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
HCL
Stars
0
Forks
1
Avg merge
24m
Merged PRs (30d)
1

Description

Fully functional Medallion Architecture (Bronze/Silver/Gold) pipelines ingesting core mines data into Fabric. Data Gateway and ExpressRoute fully established. Data shareability is established (ability to share data the data to outside of BC government). he current tools for integrating mines data are unable to support reliable analytics, leading analysts to rely on manual processes and time-consuming workarounds. Data products (reports, dashboards, etc.) break when data sources are updated, resulting in outdated information and potential errors. As a result, trust in mines divisions data is reduced, decision-making is slowed and transparency with regulatory agencies and external partners is impacted. (Source: ED Briefing Note crafted by Mia Bautista in March 2024.) Thus, The Mines Data Repository project will move this to a modern, cloud-based architecture on Microsoft Azure and Microsoft Fabric — creating a single, governed, and shareable platform for all mines data.

**Story Points:** 8

Contributor guide

No contributing guide indexed for this repository

Research direction

No repository files, tests, or entry points are named. Start by locating the existing Mines Data Platform ingestion configuration and deployment entry points, then clarify the required Bronze, Silver, and Gold pipelines, gateway and ExpressRoute setup, and data-sharing criteria. Done means the on-premise mines data is reliably ingested into Fabric and the resulting data products remain usable when sources change.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.