backstage / backstage/community-plugins
🚀 tech-insights: Incremental Fact Retriever
- Dominant language
- TypeScript
- Stars
- 422
- Forks
- 697
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 286
Description
### Workspace
tech-insights
### 🔖 Feature description
Incremental fact retrieval for Tech Insights, similar to large catalog ingestion in the incremental provider.
### 🎤 Context
In larger environments, fact retrieval can become a long-running, all-or-nothing operation. That creates a few challenges:
- Large bursts of work can cause timeouts or memory pressure.
- A single failed batch can mean retrying the entire run.
- External systems/APIs (CMDBs, monitoring) often need slower, batched access.
An incremental model would allow:
- Batch-based processing
- Progress tracking across scheduler ticks
- Partial success to be preserved
- More predictable resource usage
Early incremental fact retriever prototype highlights a gap: there is no backend service for inserting facts. Webhooks or event-based pushes from external systems could create facts on the fly, but that still needs a backend service that exposes an insert-facts function.
### ✌️ Possible Implementation
If the incremental fact retriever is a 'good' idea there are 2 tasks that would need to be done:
- Expose a backend service to insert facts for an entity. It could be used by an incremental retriever or webhooks events from external systems.
- Extend the incremental provider to function as an incremental fact retriever
### 👀 Have you spent some time to check if this feature request has been raised before?
- [x] I checked and didn't find similar issue
### 🏢 Have you read the Code of Conduct?
- [x] I have read the [Code of Conduct](https://github.com/backstage/community-plugins/blob/main/CODE_OF_CONDUCT.md)
### Are you willing to submit PR?
Yes I am willing to submit a PR!
Contributor guide
Research direction
The issue names no files, tests, or entry points. Start by locating the tech-insights incremental provider, existing fact retrieval, and backend services, then clarify whether the work includes fact insertion, incremental retrieval, or both. Done should include agreed scope and acceptance criteria for batching, progress, and partial success.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100