databrickslabs / databrickslabs/ontos

[Feature]: Support industry regulatory compliance checks (pre-set templates)

Open
#37 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

scope/compliance type/feature
Dominant language
Python
Stars
212
Forks
71
Avg merge
4d 10h
Merged PRs (30d)
43

Description

Is there an existing issue for this?
  • I have searched the existing issues
Problem statement

For specific industry regulations we need pre-set complaince checks.

Proposed Solution

Implement a repository of pre-built compliance checks that can be imported into the app. For example, for GDPR we have to ensure data is accessed with reason and expiration time. Here some ideas:

Implementable Compliance Checks for Ontos

  1. GDPR (General Data Protection Regulation)

✅ Implementable Checks:

  1. Access Grant Expiration Dates
    - Verify all grants to PII tables have expiration dates via tags
    - Check: HAS_TAG('access_expiration') for tables tagged with contains_pii
  2. Data Minimization Verification
    - Ensure tables with PII only grant access to necessary columns
    - Check: Tables with contains_pii tag must have minimized_columns tag or justification
  3. Purpose Limitation
    - Verify data access grants include purpose documentation
    - Check: HAS_TAG('access_purpose') for sensitive data
  4. Lawful Basis Documentation
    - Ensure tables with PII have documented lawful basis
    - Check: TAG('lawful_basis') IN ['consent', 'contract', 'legal_obligation', 'vital_interests', 'public_task', 'legitimate_interests']
  5. Data Retention Policies
    - Verify retention periods are defined
    - Check: HAS_TAG('retention_period') for all production tables
  6. Right to Erasure Support
    - Ensure tables with PII are tagged as supporting deletion
    - Check: TAG('supports_deletion') = 'true' for PII tables
  7. Data Portability Support
    - Verify export capability is documented
    - Check: TAG('supports_export') = 'true' for customer data
  8. Cross-Border Transfer Documentation
    - Check international storage locations are documented
    - Check: Tables in non-EU catalogs with EU data have transfer_mechanism tag
  9. Data Processing Agreement Tracking
    - Verify third-party processor agreements are documented
    - Check: Schemas shared with external groups have processor_agreement_id tag
  10. Consent Tracking
    - Ensure consent-based data has consent tracking table references
    - Check: TAG('consent_table') exists for consent-based PII

❌ Cannot Implement:

  • DPIA (requires manual assessment)
  • Actual breach detection (requires monitoring tools)
  • Real-time 72-hour breach notification (requires incident management system)

  1. CCPA/CPRA (California Consumer Privacy Act)

✅ Implementable Checks:

  1. Data Sale/Share Tracking
    - Verify tables shared externally are tagged
    - Check: TAG('data_sharing') IN ['sale', 'share', 'internal'] for all tables
  2. Purpose Disclosure
    - Ensure collection purposes are documented
    - Check: HAS_TAG('collection_purpose') for customer data tables
  3. Third-Party Disclosure Tracking
    - Verify third-party recipients are documented
    - Check: Schemas with external access have third_party_recipients tag
  4. Sensitive Personal Information Handling
    - Identify and tag sensitive PI per CPRA definition
    - Check: TAG('data_sensitivity') IN ['public', 'internal', 'confidential', 'sensitive_pi']
  5. Data Retention Limits
    - Verify maximum retention periods are defined
    - Check: HAS_TAG('max_retention_days')
  6. Consumer Request Capability
    - Ensure tables support deletion and export
    - Check: TAG('supports_dsar') = 'true' for customer tables

❌ Cannot Implement:

  • Opt-out mechanism enforcement (application logic)
  • 45-day response tracking (requires ITSM integration)

  1. HIPAA (Health Insurance Portability and Accountability Act)

✅ Implementable Checks:

  1. Minimum Necessary Access
    - Verify PHI tables have access justifications
    - Check: Grants to PHI tables have access_justification tag
  2. Audit Trail Requirements
    - Ensure PHI tables have audit logging enabled
    - Check: TAG('audit_enabled') = 'true' for PHI tables
  3. Authorization Tracking
    - Verify patient authorization is documented
    - Check: HAS_TAG('authorization_ref') for PHI tables
  4. Emergency Access Documentation
    - Ensure emergency access procedures are documented
    - Check: HAS_TAG('emergency_access_procedure') for PHI schemas
  5. Access Termination
    - Verify access grants have end dates
    - Check: HAS_TAG('access_expiration') for PHI tables
  6. De-identification Tracking
    - Mark de-identified datasets
    - Check: PHI tables must have TAG('phi_status') IN ['identified', 'deidentified', 'limited_dataset']
  7. Business Associate Agreement (BAA) Tracking
    - Verify BAAs for external data sharing
    - Check: External schemas have baa_agreement_id tag

❌ Cannot Implement:

  • Encryption verification (infrastructure level)
  • Physical safeguards (not data-level)
  • Workforce training tracking (HR system)
  • Actual breach notification (requires incident system)

  1. SOX (Sarbanes-Oxley Act)

✅ Implementable Checks:

  1. Segregation of Duties (SoD)
    - Detect conflicting access combinations on financial tables
    - Check: Users with both write and approve privileges on same financial data
  2. Access Certification Requirements
    - Verify access reviews are documented
    - Check: TAG('last_access_review') within last 90 days for financial tables
  3. Change Management Documentation
    - Ensure schema changes have approval tracking
    - Check: HAS_TAG('change_approval_id') for production financial schemas
  4. Financial Data Lineage
    - Verify lineage documentation exists
    - Check: Financial tables have HAS_TAG('data_lineage') or HAS_TAG('source_system')
  5. Access Revocation Tracking
    - Ensure terminated user access is logged
    - Check: Access grants have revocation_date when status changes
  6. Data Integrity Controls
    - Verify write access restrictions on financial tables
    - Check: Financial tables have TAG('write_protected') = 'true' or limited write grants
  7. Privileged Access Monitoring
    - Identify and track privileged users
    - Check: Admin access to financial data has TAG('privileged_access_justification')
  8. IT General Controls (ITGC)
    - Verify control documentation for critical systems
    - Check: Production catalogs have TAG('itgc_control_id')

❌ Cannot Implement:

  • Actual audit log immutability (storage level)
  • Disaster recovery testing (operational)

  1. PCI DSS (Payment Card Industry Data Security Standard)

✅ Implementable Checks:

  1. Cardholder Data Environment (CDE) Identification
    - Mark and restrict CDE tables
    - Check: TAG('pci_scope') = 'cde' for cardholder data
  2. Unique User ID Requirements
    - Verify no shared accounts have CDE access
    - Check: CDE tables do not have group-level write access
  3. Access Log Retention
    - Verify audit retention tags (1 year minimum)
    - Check: TAG('audit_retention_days') >= '365' for CDE tables
  4. Quarterly Access Reviews
    - Ensure regular review schedule
    - Check: TAG('last_access_review') within last 90 days for CDE
  5. Access Revocation for Terminated Users
    - Verify immediate revocation documented
    - Check: CDE access grants have expiration dates
  6. Data Retention Policies
    - Verify no storage beyond business need
    - Check: HAS_TAG('retention_period') and TAG('retention_justification') for CDE
  7. PAN Masking Requirements
    - Ensure masking is documented
    - Check: TAG('pan_masked') = 'true' for CDE tables or views

❌ Cannot Implement:

  • Encryption verification (infrastructure level)
  • Multi-factor authentication (auth system)
  • Vulnerability scanning (security tools)
  • Physical access controls (facility level)

  1. SOC 2 (Service Organization Control 2)

✅ Implementable Checks:

  1. Security - Access Controls
    - Verify access restrictions are documented
    - Check: All production tables have defined ownership and access policies
  2. Confidentiality - Data Classification
    - Ensure proper classification tags
    - Check: TAG('data_classification') IN ['public', 'internal', 'confidential', 'restricted']
  3. Privacy - Collection Notice Alignment
    - Verify privacy notice references
    - Check: Customer data tables have privacy_notice_version tag
  4. Processing Integrity - Data Quality
    - Ensure quality metrics are tracked
    - Check: Critical tables have TAG('quality_score') or TAG('data_quality_checks')
  5. Availability - Asset Criticality
    - Identify and tag critical assets
    - Check: Production tables have TAG('criticality') IN ['low', 'medium', 'high', 'critical']
  6. Change Management
    - Track change approvals
    - Check: Production schemas have last_change_approval tag
  7. Vendor Management
    - Document third-party data sharing
    - Check: External access has vendor_name and vendor_risk_assessment tags

❌ Cannot Implement:

  • System uptime monitoring (infrastructure)
  • Disaster recovery testing (operational)
  • Incident response execution (security tools)

  1. GLBA (Gramm-Leach-Bliley Act)

✅ Implementable Checks:

  1. NPI Access Restrictions
    - Verify restricted access to nonpublic personal information
    - Check: TAG('contains_npi') = 'true' tables have limited access grants
  2. Privacy Notice References
    - Ensure privacy notices are documented
    - Check: NPI tables have privacy_notice_url or privacy_notice_version tag
  3. Information Sharing Tracking
    - Document data sharing arrangements
    - Check: HAS_TAG('sharing_arrangement') for NPI tables
  4. Opt-Out Support
    - Verify opt-out capability is documented
    - Check: TAG('supports_optout') = 'true' for shared NPI
  5. Service Provider Oversight
    - Track third-party processors
    - Check: External schemas have service_provider_agreement tag
  6. Disposal Procedures
    - Document secure disposal capability
    - Check: HAS_TAG('disposal_procedure') for NPI tables

❌ Cannot Implement:

  • Information security program audit (organizational)
  • Pretexting protections (application/process level)

  1. FERPA (Family Educational Rights and Privacy Act)

✅ Implementable Checks:

  1. Legitimate Educational Interest
    - Verify access justifications
    - Check: Education record tables have access_justification tag for all grants
  2. Consent Tracking
    - Document consent for non-exception disclosures
    - Check: HAS_TAG('disclosure_consent') or TAG('disclosure_exception') for education records
  3. Directory Information Opt-Out
    - Mark directory information tables
    - Check: TAG('directory_info') = 'true' and TAG('supports_optout') = 'true'
  4. Access Logging Requirements
    - Ensure audit trails exist
    - Check: TAG('audit_enabled') = 'true' for education records
  5. Third-Party Disclosure Tracking
    - Document external disclosures
    - Check: HAS_TAG('disclosure_log_table') for education records
  6. Amendment Request Support
    - Verify update capability
    - Check: TAG('supports_amendments') = 'true' for student records

❌ Cannot Implement:

  • Annual notification process (organizational)

  1. LGPD (Lei Geral de Proteção de Dados) - Brazil

✅ Implementable Checks:

  • Similar to GDPR - reuse GDPR checks with LGPD-specific adjustments
  • Lawful basis (10 legal bases): TAG('lgpd_legal_basis')
  • Data Protection Agent designation: TAG('data_protection_agent')
  • International transfer safeguards: TAG('international_transfer_mechanism')

  1. PIPEDA (Canada)

✅ Implementable Checks:

  • Similar to GDPR/CCPA - reuse consent and purpose checks
  • Consent documentation: TAG('pipeda_consent')
  • Purpose identification: TAG('collection_purpose')
  • Safeguards documentation: TAG('safeguard_level')
  • Cross-border transfer: TAG('cross_border_transfer_doc')

  1. FINRA/SEC Regulations

✅ Implementable Checks:

  1. Books and Records Retention
    - Verify immutable storage tags (SEC Rule 17a-4)
    - Check: Financial records have TAG('immutable_storage') = 'true'
  2. Record Retention Periods
    - Ensure minimum retention (6 years for most records)
    - Check: TAG('retention_period_years') >= '6' for financial records
  3. Segregation of Duties - Trading
    - Detect conflicting trading system access
    - Check: No user has both trade_execution and trade_approval on same tables
  4. KYC Data Access Controls
    - Restrict Know Your Customer data access
    - Check: KYC tables have strict access controls and TAG('pii_type') = 'kyc'
  5. Supervision Documentation
    - Verify communications data lineage
    - Check: Communications tables have supervision_policy_id tag

❌ Cannot Implement:

  • Best execution monitoring (trading system)
  • Insider trading surveillance (requires trading data analysis)
  • Market manipulation detection (complex analytics)

  1. BCBS 239 (Basel Committee - Banking)

✅ Implementable Checks:

  1. Data Lineage Tracking
    - Ensure lineage is documented
    - Check: Risk data tables have HAS_TAG('data_lineage') or lineage_documented = true
  2. Data Quality Metrics
    - Verify quality scoring exists
    - Check: Risk tables have TAG('accuracy_score'), TAG('completeness_score'), TAG('timeliness_sla')
  3. Data Governance Framework
    - Ensure ownership and stewardship
    - Check: All risk data has defined owner and data steward
  4. Historical Data Retention
    - Verify historical data availability
    - Check: Risk tables have TAG('historical_retention_years')
  5. Reconciliation Controls
    - Document reconciliation processes
    - Check: Aggregated risk tables have reconciliation_table_ref tag
  6. Data Architecture Documentation
    - Ensure source system documentation
    - Check: Risk data has source_system and data_architecture_doc tags

❌ Cannot Implement:

  • Actual data quality measurement (requires profiling tools)
  • Aggregation accuracy testing (requires calculation verification)

  1. NIST Frameworks

✅ Implementable Checks:

  1. Asset Inventory
    - Verify all assets are cataloged
    - Check: All catalogs/schemas/tables have metadata and ownership
  2. Data Classification
    - Ensure classification tags exist
    - Check: HAS_TAG('data_classification')
  3. Access Control Policy Enforcement
    - Verify access policies are documented
    - Check: Sensitive data has access_policy_id tag
  4. Privacy Risk Assessment
    - Document PIA completion
    - Check: PII tables have privacy_impact_assessment_id tag
  5. Supply Chain Risk Management
    - Track third-party data sources
    - Check: External data sources have vendor_risk_rating tag

❌ Cannot Implement:

  • Anomaly detection (requires monitoring tools)
  • Incident response execution (security operations)
  • Recovery testing (operational)

  1. FedRAMP (US Government Cloud)

✅ Implementable Checks:

  1. Least Privilege Verification
    - Ensure minimal access grants
    - Check: Federal data tables have access justifications
  2. Audit Log Protection
    - Verify audit settings
    - Check: Federal data has TAG('audit_immutable') = 'true'
  3. Separation of Duties
    - Detect SoD violations
    - Check: Conflicting access combinations on federal data
  4. Security Assessment Documentation
    - Track assessment dates
    - Check: Federal catalogs have last_security_assessment tag

❌ Cannot Implement:

  • Continuous monitoring (requires SIEM)
  • Boundary protection (network level)
  • Multi-factor authentication (auth system)

Cross-Cutting Checks (Universal)

Access Management

  1. ✅ Role-based access verification
  2. ✅ Access grant justification requirements
  3. ✅ Time-bound access (expiration dates)
  4. ✅ Access revocation tracking
  5. ✅ Least privilege verification
  6. ✅ Privileged access documentation
  7. ✅ Emergency access procedures documented
  8. ✅ Regular access reviews/recertification tracking

Audit & Compliance

  1. ✅ Audit logging enabled tags
  2. ✅ Audit log retention period definition
  3. ✅ Access attempt logging tags

Data Governance

  1. ✅ Data ownership requirements
  2. ✅ Data stewardship assignment
  3. ✅ Retention policy definition
  4. ✅ Data lineage documentation
  5. ✅ Data quality metrics tracking
  6. ✅ Metadata completeness (descriptions/comments)
  7. ✅ Data catalog completeness

Data Protection

  1. ✅ Data classification tagging
  2. ✅ Encryption tags (documenting encryption status)
  3. ✅ Masking/anonymization tags
  4. ✅ Sensitive data identification

Vendor & Third-Party

  1. ✅ Third-party access tracking
  2. ✅ Data processing agreements documented
  3. ✅ Vendor risk assessment tags
  4. ✅ Subprocessor documentation

Incident Response

  1. ✅ Incident detection capability tags
  2. ✅ Breach notification workflow tags

Implementation Priority

Tier 1 (Universal - High Impact)

  1. Data classification and sensitive data identification
  2. Access grant expiration dates
  3. Data owner assignment
  4. Retention policy definition
  5. Access justification requirements

Tier 2 (Regulation-Specific - High Compliance Value)

  1. GDPR: Lawful basis + purpose limitation
  2. HIPAA: Minimum necessary + PHI audit trails
  3. SOX: Segregation of duties
  4. PCI DSS: CDE identification + quarterly reviews
  5. CCPA: Sale/share tracking

Tier 3 (Enhanced Governance)

  1. Data lineage documentation
  2. Data quality metrics
  3. Third-party disclosure tracking
  4. Privacy impact assessments
  5. Change management documentation
Additional Context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are identified in the issue. Start by clarifying the supported compliance-check scope and locating the repository's relevant implementation and test areas; done would require an agreed, tested set of importable templates and defined boundaries for unsupported checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.