crewAIInc / crewAIInc/crewAI

Could CrewAI generate the first live SABLE reliability trace?

Open
#7,392 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

vendor-pitch
Dominant language
Python
Stars
58.8k
Forks
8.5k
Avg merge
1d 15h
Merged PRs (30d)
109

Description

Hi CrewAI maintainers — I’m building SABLE, a model-independent reliability evaluator for tool-using agents. I’m looking for one live third-party runtime submission, and CrewAI is a strong candidate because it has active tool-using agents and production-oriented execution controls.

I’m not asking you to change CrewAI or adopt SABLE. The experiment is intentionally small:

  1. Run one real CrewAI agent against a SABLE task.
  2. Capture the runtime-generated tool calls/results + before/after state hashes.
  3. Run SABLE’s deterministic replay/evaluation.
  4. Publish the GitHub Actions run/artifact as provenance.

SABLE quickstart: https://github.com/socksninja/sable-agent-reliability/blob/main/docs/EXTERNAL_SUBMISSION_QUICKSTART.md
Protocol: https://github.com/socksninja/sable-agent-reliability/blob/main/docs/TRACE_SUBMISSION_PROTOCOL_V0.9.md
Open call: https://github.com/socksninja/sable-agent-reliability/issues/17

The trace must come from CrewAI itself — no hand-authored evidence. If the integration is awkward, even one failed attempt that exposes an incompatibility is useful feedback.

This is an invitation to a small, reproducible reliability experiment, not a marketing benchmark claim.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with SABLE’s EXTERNAL_SUBMISSION_QUICKSTART.md and TRACE_SUBMISSION_PROTOCOL_V0.9.md, then identify the smallest CrewAI runtime example that can execute the SABLE task. Run one real agent, capture its tool calls, results, and before/after state hashes, and use SABLE’s deterministic evaluation. Done means a published GitHub Actions run or artifact, or a documented failed attempt showing an incompatibility.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
ai, ci-cd, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.