Could CrewAI generate the first live SABLE reliability trace?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 58.8k
- Forks
- 8.5k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 109
Description
Hi CrewAI maintainers — I’m building SABLE, a model-independent reliability evaluator for tool-using agents. I’m looking for one live third-party runtime submission, and CrewAI is a strong candidate because it has active tool-using agents and production-oriented execution controls.
I’m not asking you to change CrewAI or adopt SABLE. The experiment is intentionally small:
- Run one real CrewAI agent against a SABLE task.
- Capture the runtime-generated tool calls/results + before/after state hashes.
- Run SABLE’s deterministic replay/evaluation.
- Publish the GitHub Actions run/artifact as provenance.
SABLE quickstart: https://github.com/socksninja/sable-agent-reliability/blob/main/docs/EXTERNAL_SUBMISSION_QUICKSTART.md
Protocol: https://github.com/socksninja/sable-agent-reliability/blob/main/docs/TRACE_SUBMISSION_PROTOCOL_V0.9.md
Open call: https://github.com/socksninja/sable-agent-reliability/issues/17
The trace must come from CrewAI itself — no hand-authored evidence. If the integration is awkward, even one failed attempt that exposes an incompatibility is useful feedback.
This is an invitation to a small, reproducible reliability experiment, not a marketing benchmark claim.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with SABLE’s EXTERNAL_SUBMISSION_QUICKSTART.md and TRACE_SUBMISSION_PROTOCOL_V0.9.md, then identify the smallest CrewAI runtime example that can execute the SABLE task. Run one real agent, capture its tool calls, results, and before/after state hashes, and use SABLE’s deterministic evaluation. Done means a published GitHub Actions run or artifact, or a documented failed attempt showing an incompatibility.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python
- Domain
- ai, ci-cd, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100