aws-samples / aws-samples/sample-aiml-security-assessment
Extend checks for Security Hub CSPM AI Security Best Practices parity, plus false-Failed bug fixes (AC-04/FS-08/FS-66/FS-39/FS-41/FS-50, BR-01 link)
- Dominant language
- HTML
- Stars
- 43
- Forks
- 20
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 4
Description
**Is your feature request related to a problem? Please describe.**
This is primarily about extending the tool's checks to close control-parity gaps against **AWS Security Hub CSPM's AI Security Best Practices v1.0.0 standard** (`arn:aws:securityhub:::standards/ai-security-best-practices/v/1.0.0`, 31 controls). A gap analysis (`docs/AI_SECURITY_BEST_PRACTICES_GAP_ANALYSIS.md`) identified missing checks and severity/disposition inconsistencies relative to the published Security Hub controls; this issue tracks closing those gaps.
While doing that work and rerunning the suite against live AWS accounts (single-account and multi-account), we also found and confirmed a smaller set of pre-existing check-logic bugs that made several checks always report Failed regardless of actual account configuration, or made them structurally unable to report Passed for accounts lacking a precondition resource (e.g. no SageMaker endpoints, no Bedrock Guardrails). These are bundled into the same effort since they were found and fixed alongside the parity work:
- AC-04 (AgentCore Observability), FS-08 (AgentCore Policy Engine), FS-66 (AgentCore End-User Identity Propagation) read API fields (`loggingConfig`/`tracingConfig`/`authorizerConfiguration` off `ListAgentRuntimes`) that don't exist on those responses, so they always evaluate as unconfigured.
- FS-09 (Agent Transaction Limits), FS-52 (Bedrock SDK Version Currency), FS-67 (Agent Financial Transaction Thresholds) use broad keyword matching against Lambda function names that also matches the assessment tool's own deployed Lambdas (`aiml-security-*`), causing false self-flagging.
- FS-39 (SageMaker Clarify Bias) and FS-41 (SageMaker Clarify Explainability) check for monitoring schedules without first checking whether any SageMaker endpoint exists to be monitored, so they always fail on accounts with zero SageMaker endpoints.
- FS-50 (Guardrail Relevance Grounding) is missing the "no guardrails -> N/A" guard present on every sibling guardrail-filter check, so it fails instead of reporting N/A when zero guardrails exist.
- BR-01's reference field embeds two URLs joined by a literal newline, breaking the rendered link in the HTML report.
**Describe the solution you'd like**
- Extend the check set to cover the Security Hub CSPM AI Security Best Practices controls identified as gaps, adopting a documented Likelihood x Impact severity/disposition methodology tool-wide for consistency.
- Fix each of the check-logic bugs above to read the correct API fields / add the missing precondition checks, add an exclusion helper so self-referential keyword matching does not flag the tool's own infrastructure, and fix the BR-01 reference URL.
- Update the check documentation in `docs/SECURITY_CHECKS.md` and `docs/SECURITY_CHECKS_FINSERV.md` to reflect the new and corrected checks, and add/update unit tests for each change.
**Describe alternatives you've considered**
For the bug-fix portion, could suppress those checks entirely for accounts without the relevant resources, but that would hide the check from the report rather than correctly reporting N/A/Informational, which is the existing convention used by sibling checks (FS-16, FS-20, FS-36, FS-45, FS-47).
**Additional context**
Verified via unit tests (1116 passing) and live reruns against real single-account and multi-account AWS environments.
Contributor guide
Research direction
Start with docs/AI_SECURITY_BEST_PRACTICES_GAP_ANALYSIS.md, then review the checks for AC-04, FS-08, FS-66, FS-09, FS-52, FS-67, FS-39, FS-41, FS-50, and BR-01. Run the unit-test suite and inspect the existing sibling checks named in the issue for N/A behavior. Done means the identified parity gaps and false-Failed cases are corrected, documentation is updated, and the tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, documentation, security, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100