aws-samples / aws-samples/sample-aiml-security-assessment

Extend checks for Security Hub CSPM AI Security Best Practices parity, plus false-Failed bug fixes (AC-04/FS-08/FS-66/FS-39/FS-41/FS-50, BR-01 link)

Open
#50 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
HTML
Stars
43
Forks
20
Avg merge
2d 23h
Merged PRs (30d)
4

Description

**Is your feature request related to a problem? Please describe.**
This is primarily about extending the tool's checks to close control-parity gaps against **AWS Security Hub CSPM's AI Security Best Practices v1.0.0 standard** (`arn:aws:securityhub:::standards/ai-security-best-practices/v/1.0.0`, 31 controls). A gap analysis (`docs/AI_SECURITY_BEST_PRACTICES_GAP_ANALYSIS.md`) identified missing checks and severity/disposition inconsistencies relative to the published Security Hub controls; this issue tracks closing those gaps.

While doing that work and rerunning the suite against live AWS accounts (single-account and multi-account), we also found and confirmed a smaller set of pre-existing check-logic bugs that made several checks always report Failed regardless of actual account configuration, or made them structurally unable to report Passed for accounts lacking a precondition resource (e.g. no SageMaker endpoints, no Bedrock Guardrails). These are bundled into the same effort since they were found and fixed alongside the parity work:
- AC-04 (AgentCore Observability), FS-08 (AgentCore Policy Engine), FS-66 (AgentCore End-User Identity Propagation) read API fields (`loggingConfig`/`tracingConfig`/`authorizerConfiguration` off `ListAgentRuntimes`) that don't exist on those responses, so they always evaluate as unconfigured.
- FS-09 (Agent Transaction Limits), FS-52 (Bedrock SDK Version Currency), FS-67 (Agent Financial Transaction Thresholds) use broad keyword matching against Lambda function names that also matches the assessment tool's own deployed Lambdas (`aiml-security-*`), causing false self-flagging.
- FS-39 (SageMaker Clarify Bias) and FS-41 (SageMaker Clarify Explainability) check for monitoring schedules without first checking whether any SageMaker endpoint exists to be monitored, so they always fail on accounts with zero SageMaker endpoints.
- FS-50 (Guardrail Relevance Grounding) is missing the "no guardrails -> N/A" guard present on every sibling guardrail-filter check, so it fails instead of reporting N/A when zero guardrails exist.
- BR-01's reference field embeds two URLs joined by a literal newline, breaking the rendered link in the HTML report.

**Describe the solution you'd like**
- Extend the check set to cover the Security Hub CSPM AI Security Best Practices controls identified as gaps, adopting a documented Likelihood x Impact severity/disposition methodology tool-wide for consistency.
- Fix each of the check-logic bugs above to read the correct API fields / add the missing precondition checks, add an exclusion helper so self-referential keyword matching does not flag the tool's own infrastructure, and fix the BR-01 reference URL.
- Update the check documentation in `docs/SECURITY_CHECKS.md` and `docs/SECURITY_CHECKS_FINSERV.md` to reflect the new and corrected checks, and add/update unit tests for each change.

**Describe alternatives you've considered**
For the bug-fix portion, could suppress those checks entirely for accounts without the relevant resources, but that would hide the check from the report rather than correctly reporting N/A/Informational, which is the existing convention used by sibling checks (FS-16, FS-20, FS-36, FS-45, FS-47).

**Additional context**
Verified via unit tests (1116 passing) and live reruns against real single-account and multi-account AWS environments.

Contributor guide

Open the contributing guide

Research direction

Start with docs/AI_SECURITY_BEST_PRACTICES_GAP_ANALYSIS.md, then review the checks for AC-04, FS-08, FS-66, FS-09, FS-52, FS-67, FS-39, FS-41, FS-50, and BR-01. Run the unit-test suite and inspect the existing sibling checks named in the issue for N/A behavior. Done means the identified parity gaps and false-Failed cases are corrected, documentation is updated, and the tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, documentation, security, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.