aws-samples / aws-samples/sample-collaborative-ai-dlc

[RFC]: Autonomous Tracker-Driven Intent Execution

Open
#415 0 comments 0 reactions 0 assignees View on GitHub
rfc
Dominant language
JavaScript
Stars
75
Forks
23
Avg merge
3d 12h
Merged PRs (30d)
30

Description

### Summary

Add opt-in, tracker-driven autonomous intent execution to Collaborative AI-DLC. When a configured Jira Cloud, GitHub Issues, or GitLab Issues binding receives a newly created work item that matches project filters, the platform verifies and normalizes the webhook, creates an intent from the work item, selects or composes the appropriate workflow scope, validates it, and starts execution automatically.

Each project chooses one of three autonomy modes: **`fully-autonomous`, `semi-autonomous`, or `partially-autonomous`**. These modes control how workflow decisions, approval gates, and clarification questions are handled.

Existing pull/merge request behavior remains unchanged. Collaborative AI-DLC must not approve or merge its own pull/merge requests.

This proposal moves the platform toward a dark-factory operating model while deliberately retaining the existing human PR boundary. A more precise description is **lights-out execution until the existing PR workflow.**

### Motivation

Collaborative AI-DLC currently supports:

- Jira Cloud, GitHub Issues, and GitLab Issues tracker bindings;
- manual import of a tracker work item into an intent;
- tracker provenance on the resulting intent;
- named workflow scopes and validated custom execution grids;
- a composer that can propose a named or custom scope;
- durable execution, questions, validation gates, steering, and PR creation.

However, a person must still discover the work item, create the intent, apply the scope proposal, start the run, watch for gates, and locate the correct stakeholder when input is required. This manual coordination prevents the platform from operating continuously from an existing engineering backlog.

The target experience is:
> [!IMPORTANT]
> A stakeholder creates a suitably classified work item in the team's existing tracker. Collaborative AI-DLC starts the right workflow automatically, proceeds according to the project's autonomy policy, asks only the people whose input is required, and reaches the existing PR workflow without routine operator intervention.

The design adopts event-driven intake, workflow-level orchestration, constrained agent permissions, deterministic validation, durable state, and progressive autonomy. It does not introduce autonomous PR approval, merge, or deployment.

References:
- [What Is a Dark Factory? The Concept of Fully Autonomous AI-Driven Codebases](https://www.mindstudio.ai/blog/what-is-dark-factory-autonomous-ai-codebase)
- [What Is a Dark Factory? The AI Coding Pattern That Ships Code Without Human Review](https://www.mindstudio.ai/blog/what-is-a-dark-factory-ai-coding)

### Goals

- Start eligible intents automatically from tracker work-item creation events.
- Support Jira Cloud, GitHub Issues, and GitLab Issues.
- Filter automatic starts by work-item type and conventional title.
- Select or generate the execution scope automatically.
- Provide three explicit levels of autonomy.
- Route questions and gates to the relevant project persona.
- Make every intake event and autonomous decision observable and auditable.
- Preserve the existing PR behavior and human PR boundary.

### Non-Goals (at least in a first version)

- Reacting to edits made after an intent starts.
- Automatically consuming new tracker comments.
- Changing the existing review-feedback mechanism.
- Approving or merging pull/merge requests.
- Deploying changes automatically.
- Replacing existing manual intent creation or tracker import.

### Detailed design

### 1. Project Automation Policy

Automation is configured per project and tracker binding. It is disabled by default.

Conceptual configuration:

```json
{
"enabled": true,
"events": ["created"],
"filters": {
"workTypes": ["feat", "fix", "docs", "rfc"],
"titlePatterns": ["^(feat|fix|docs|rfc)(\\(.+\\))?!?:\\s.+"],
"aliases": {
"feature": "feat",
"bug": "fix",
"issue": "fix",
"doc": "docs"
}
},
"autonomyMode": "semi-autonomous",
"scopePolicy": {
"mode": "composer",
"allowCustomScope": true
},
"fallbackPersonas": ["Product Owner"]
}
```

The initial normalized work-type vocabulary is:

- `feat`
- `fix`
- `docs`
- `rfc`
- `refactor`
- `perf`
- `test`
- `chore`

Projects select the types that may launch automatically and can define aliases such as `feature`, `bug`, `issue`, or `doc`.

Provider matching uses the strongest available signal:

- **Jira Cloud:** issue type and optional conventional title prefix.
- **GitHub Issues:** issue type when available, labels, and conventional title prefix.
- **GitLab Issues:** issue type when available, labels, and conventional title prefix.

The first version must support filtering by normalized issue type and title. Additional provider-native filters can be added without changing the normalized intake contract.

### 2. Webhook Intake

Each provider sends work-item creation events to a provider-specific webhook endpoint.

The intake path:

1. Verify the provider signature or shared secret before processing the event.
2. Reject unsupported event types and oversized payloads.
3. Normalize the event into a provider-neutral `TrackerWorkItemCreated` event.
4. Resolve the tracker binding and Collaborative AI-DLC project.
5. Deduplicate using provider, provider instance, work-item ID, action, and delivery ID.
6. Persist an intake record before dispatching asynchronous processing.
7. Evaluate the project's automation filters.
8. Ignore unmatched work items with an auditable reason.
9. Enqueue matched work items for intent creation.

Conceptual normalized event:

```json
{
"eventId": "provider-delivery-id",
"provider": "jira-cloud|github-issues|gitlab-issues",
"bindingId": "binding-id",
"resourceType": "issue",
"resourceId": "ABC-123",
"resourceUrl": "https://...",
"title": "feat(auth): add passkeys",
"workType": "feat",
"occurredAt": "2026-08-20T10:00:00Z"
}
```

Webhook handling must be retry-safe. Re-delivery of the same event or concurrent deliveries for the same work item must not create multiple intents.

### 3. System-Created Intent

The automation worker fetches the canonical work item through the existing tracker abstraction rather than trusting the webhook payload as the complete prompt.

It creates an intent with:

- the work-item title;
- a prompt based on the current title and description;
- tracker source provenance;
- the project's pinned workflow and current runtime settings;
- an auditable actor such as `system:tracker-automation`;
- a link from the intake record to the intent.

The first version takes a snapshot at creation. Later edits do not mutate a running intent. Processing comments added after launch is deferred.

### 4. Automatic Scope Composition

Scope selection is automatic for tracker-created intents.

The composer must be hardened before unattended application:

1. Match an existing named scope when it accurately represents the work.
2. Otherwise generate a custom `EXECUTE`/`SKIP` grid using the existing composed-grid capability.
3. Validate every proposal with the authoritative execution-plan validator.
4. If validation fails, provide the structured errors to the composer and allow a bounded repair attempt.
5. Persist the proposal, rationale, confidence, validation result, model/version, and whether a named or custom scope was selected.
6. Apply a valid proposal according to the configured autonomy mode.

No model output can bypass plan validation. Confidence is advisory and must be calibrated through evaluation rather than treated as proof of correctness.

Before enabling automatic launch broadly, create a composer evaluation suite from representative historical and synthetic work items. At minimum, measure:

- valid-plan rate;
- named-scope selection accuracy;
- stage precision and recall for custom scopes against expert-reviewed grids;
- confidence calibration;
- frequency of human recomposition or rewind;
- downstream question, failure, and PR rework rates;
- results by work type and project workflow.

The evaluation must identify work types or workflows that are not ready for automatic composition. Automation can then be enabled progressively for proven combinations rather than globally.

### 5. Autonomy Modes

The selected mode is snapshotted on the intent so a later project-setting change does not alter a running execution.

| Mode | Scope and plan | Questions | Approval and validation gates |
|--------------------------|---------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------|
| **Fully autonomous** | The composer selects, validates, and applies the scope automatically. | The system makes bounded decisions from the issue, project rules, artifacts, and prior decisions. It halts only when no valid continuation exists, a capability is unavailable, or policy forbids an autonomous choice. | Routine gates are resolved automatically according to policy. Hard operational failures still halt and notify. |
| **Semi-autonomous** | The composer selects, validates, and applies the scope automatically. Valid plans are approved automatically. | The agent may ask targeted questions when information required for a defensible decision is missing. | Routine plan and validation gates are resolved automatically. Questions and exceptional halt-and-ask gates remain human. |
| **Partially autonomous** | The composer proposes the scope, but uncertainty or configured checkpoints can require review. | The agent asks when uncertain or when a configured persona owns the decision. | Existing human approvals remain for configured stages and decisions. |

Autonomy is a policy applied to the same workflow, not three separate workflow definitions.

Every automatic decision records:

- the gate or question resolved;
- the selected answer;
- the policy and autonomy mode that permitted it;
- the evidence or artifact references used;
- the model and runtime identity when an agent made the decision;
- the timestamp and resulting state transition.

The platform must not fabricate a human identity for an automatic answer.

### 6. Persona-Aware Escalation

Questions and gates must target the person responsible for the decision, not a generic project audience.

This depends on project-level personas from [#323](https://github.com/aws-samples/sample-collaborative-ai-dlc/issues/323) and reliable multi-channel notifications from [#289](https://github.com/aws-samples/sample-collaborative-ai-dlc/issues/289).

Initial personas may include:

- Product Owner;
- Business Analyst;
- Architect;
- Developer;
- Tester/QA;
- Security/Compliance;
- Operations/Platform Engineer.

Stages, question types, or workflow rules declare one or more target personas. The notification dispatcher resolves those personas to project members and sends a deep link to the exact gate.

Routing fallback:

1. Explicitly configured persona for the gate or stage.
2. Workflow or stage default persona.
3. Project automation fallback persona.
4. Project owner.

If multiple members hold the persona, the first version notifies all of them. Answering the gate closes it for the other recipients. If no recipient can be resolved, the intent remains safely parked and the missing routing configuration is surfaced to the project owner or administrator.

The core webhook and execution work can be developed independently, but persona-targeted autonomous operation is not complete until the notification and persona capabilities are available. A project-owner fallback can support staged rollout.

### 7. Intent Lifecycle

```text
Tracker work item created
-> webhook verified and normalized
-> binding and filters evaluated
-> intake event deduplicated
-> intent created from canonical tracker data
-> composer selects or generates scope
-> execution plan validated
-> autonomy policy evaluated
-> intent starts automatically
-> workflow runs
-> automatic decision, or
-> targeted question/gate and notification
-> existing PR behavior
```

If scope composition cannot produce a valid plan:

- fully autonomous mode records a blocked intake or intent and notifies the configured fallback recipient;
- semi-autonomous mode opens a targeted scope-resolution question;
- partially autonomous mode presents the proposal and validation errors for review.

An operator can disable a rule, replay a failed intake event, or manually start a blocked intent after correction.

### 8. Pull/Merge Request Boundary

This RFC does not change the current PR system.

- Existing `intent-pr` and `pr-per-unit` strategies remain as implemented.
- Existing branch, commit, push, readiness, feedback, and PR creation behavior remains unchanged.
- Collaborative AI-DLC does not approve its own PR.
- This RFC adds no automatic merge behavior.
- This RFC adds no automatic deployment behavior.

The autonomy mode affects execution decisions before and during workflow execution. It does not weaken the existing source-control boundary.

### 9. Observability and Controls

Add an automation view, or equivalent project-level observability, showing:

- received, matched, ignored, started, blocked, and failed intake events;
- the filter rule and work type that matched;
- the linked tracker item and intent;
- the composer proposal and validation result;
- the snapshotted autonomy mode;
- automatic decisions and human escalations;
- failure reason and replay action.

Required controls:

- automation disabled by default;
- project owner or administrator configuration only;
- per-binding kill switch;
- per-work-type allowlist;
- replay without duplicate intent creation;
- rate and concurrency limits;
- audit history for configuration changes;
- no silent fallback from an invalid custom scope to a broader scope.

### 10. Security and Reliability

- Verify every webhook using provider-supported signatures or secrets.
- Store webhook secrets in the existing secret-management boundary.
- Use least-privilege credentials from the resolved binding.
- Fetch canonical work-item content after verification.
- Redact provider payloads and credentials from logs.
- Bound retries and send exhausted events to a recoverable failure state or dead-letter path.
- Enforce idempotency at the persistence layer.
- Attribute all system actions explicitly and preserve the source event.
- Apply existing prompt-injection and workspace/tool boundaries to tracker content.

### Alternatives considered

### Poll Trackers Instead of Webhooks

Polling is simpler for some installations but increases latency, provider API usage, and reconciliation complexity. It may be offered as a fallback where webhooks cannot be configured, but webhooks should be the primary design.

### Map Each Work Type to One Fixed Named Scope

This is deterministic but too coarse. Two `feat` issues can require different stages, and a fixed mapping cannot use validated custom grids. Work type should decide whether automation is allowed; the composer should decide the actual scope.

### Require Human Review for Every Imported Intent

This preserves the current model but does not materially reduce coordination cost. Partially autonomous mode remains available for teams that require these checkpoints.

### Separate Workflows for Each Autonomy Level

Duplicating workflows would create configuration drift. Autonomy should be a snapshotted execution policy over one workflow.

### Build Provider-Specific Automation Pipelines

Provider-specific ingestion is necessary, but downstream intent creation must use one normalized event and tracker abstraction. Otherwise Jira, GitHub, and GitLab behavior will diverge.

### Automatically Approve, Merge, or Deploy

Rejected for this proposal. It changes the trust and security boundary and conflicts with the requirement that Collaborative AI-DLC never approve its own PR.

### Breaking changes

None for existing projects.

- Tracker automation is additive and disabled by default.
- Existing manual intent creation and tracker import continue to work.
- Existing workflows and scopes continue to work.
- Existing PR strategies and behavior are unchanged.
- Existing projects without personas or notification preferences fall back to current project ownership during rollout.

The normalized autonomy-mode names may eventually supersede the narrower construction-only `gated`/`autonomous` setting. Migration must preserve current behavior and snapshot the resolved policy on new intents.

### Open questions

1. Should the conventional work-type vocabulary be platform-fixed with configurable aliases, or fully tenant-configurable?
2. For GitHub and GitLab, what precedence applies when issue type, label, and title prefix disagree?
3. What composer evaluation thresholds are required before enabling semi-autonomous or fully autonomous launch for a workflow and work-type pair?
4. In fully autonomous mode, which decisions remain non-delegable besides the existing PR boundary, missing capabilities, and explicit policy restrictions?
5. Should multiple users with the same persona all be eligible to answer, or should projects support a primary or on-call assignee? => #289 #323
6. How long should blocked intake events remain replayable?
7. Should a later version consume new tracker comments automatically, or continue requiring explicit human selection and bounded feedback batches?
8. When https://github.com/awslabs/aidlc-workflows/issues/801 will be implemented, we could imagine a "code-review"-only scope that is triggered when a PR is created.

Contributor guide

Open the contributing guide

Research direction

Read the existing tracker abstraction, provider-specific webhook endpoints, composer, and authoritative execution-plan validator mentioned in the proposal. Map how intake, intent creation, scope validation, durable execution, and audit records currently connect; done means a reviewed implementation covering safe event handling, autonomy policies, and the existing human PR boundary.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, gitlab, javascript
Domain
backend-api-design, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.