bcgov / bcgov/Mines-Data-Platform
Production Ingestion Pipelines for Fabric to load on premise data
- Dominant language
- HCL
- Stars
- 0
- Forks
- 1
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
Fully functional Medallion Architecture (Bronze/Silver/Gold) pipelines ingesting core mines data into Fabric. Data Gateway and ExpressRoute fully established. Data shareability is established (ability to share data the data to outside of BC government). he current tools for integrating mines data are unable to support reliable analytics, leading analysts to rely on manual processes and time-consuming workarounds. Data products (reports, dashboards, etc.) break when data sources are updated, resulting in outdated information and potential errors. As a result, trust in mines divisions data is reduced, decision-making is slowed and transparency with regulatory agencies and external partners is impacted. (Source: ED Briefing Note crafted by Mia Bautista in March 2024.) Thus, The Mines Data Repository project will move this to a modern, cloud-based architecture on Microsoft Azure and Microsoft Fabric — creating a single, governed, and shareable platform for all mines data.
**Story Points:** 8
Contributor guide
No contributing guide indexed for this repository
Research direction
No repository files, tests, or entry points are named. Start by locating the existing Mines Data Platform ingestion configuration and deployment entry points, then clarify the required Bronze, Silver, and Gold pipelines, gateway and ExpressRoute setup, and data-sharing criteria. Done means the on-premise mines data is reliably ingested into Fabric and the resulting data products remain usable when sources change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100