backstage / backstage/community-plugins

🚀 tech-insights: Incremental Fact Retriever

Open
#7,574 13 comments 3 reactions 0 assignees View on GitHub
enhancement workspace/tech-insights
Dominant language
TypeScript
Stars
422
Forks
697
Avg merge
2d 6h
Merged PRs (30d)
286

Description

### Workspace

tech-insights

### 🔖 Feature description

Incremental fact retrieval for Tech Insights, similar to large catalog ingestion in the incremental provider.

### 🎤 Context

In larger environments, fact retrieval can become a long-running, all-or-nothing operation. That creates a few challenges:

- Large bursts of work can cause timeouts or memory pressure.
- A single failed batch can mean retrying the entire run.
- External systems/APIs (CMDBs, monitoring) often need slower, batched access.

An incremental model would allow:

- Batch-based processing
- Progress tracking across scheduler ticks
- Partial success to be preserved
- More predictable resource usage

Early incremental fact retriever prototype highlights a gap: there is no backend service for inserting facts. Webhooks or event-based pushes from external systems could create facts on the fly, but that still needs a backend service that exposes an insert-facts function.

### ✌️ Possible Implementation

If the incremental fact retriever is a 'good' idea there are 2 tasks that would need to be done:
- Expose a backend service to insert facts for an entity. It could be used by an incremental retriever or webhooks events from external systems.
- Extend the incremental provider to function as an incremental fact retriever

### 👀 Have you spent some time to check if this feature request has been raised before?

- [x] I checked and didn't find similar issue

### 🏢 Have you read the Code of Conduct?

- [x] I have read the [Code of Conduct](https://github.com/backstage/community-plugins/blob/main/CODE_OF_CONDUCT.md)

### Are you willing to submit PR?

Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by locating the tech-insights incremental provider, existing fact retrieval, and backend services, then clarify whether the work includes fact insertion, incremental retrieval, or both. Done should include agreed scope and acceptance criteria for batching, progress, and partial success.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.