[TASK] Regression Testing for dotAI Workflows Features

Description

Complete regression testing of the dotAI Workflows feature, which adds five AI-powered workflow sub-actions (actionlets) to dotCMS that integrate AI capabilities into content workflows:

1. AI Auto-Tag Content - Automatically generates tags based on content
2. AI Content Prompt - Generates text content from custom prompts
3. AI Embeddings - Controls semantic search indexing
4. AI Image Auto-Tagging and Descriptions - Generates alt text and metadata from images
5. AI Translate Content - Performs multi-language translation

Reference Documentation: https://dev.dotcms.com/docs/dotai-workflows

---

Acceptance Criteria

1. AI Auto-Tag Content Actionlet

Configuration & Parameters

• Verify "Overwrite tags" option (true/false) works correctly
• Test "Limit keywords to pre-existing tags" option (true/false)
• Validate "Update content asynchronously" integer value (seconds)
• Test AI model selection dropdown
• Verify Temperature setting range (0.1-2.0)
• Validate parameter validation and error messages

Functionality

• Test automatic tag generation from content values
• Verify tags are assigned to tag fields when present
• Test tag field population with generated tags
• Verify overwrite behavior when tags already exist
• Test creation of new tags vs limiting to existing tags
• Validate async execution timing matches configured seconds
• Test with different content types and field configurations
• Verify proper permissions for tag creation

Edge Cases

• Test with empty content
• Test with content containing no taggable information
• Test with multiple tag fields in same content
• Verify behavior when tag field is read-only
• Test concurrent tagging operations

---

2. AI Content Prompt Actionlet

Configuration & Parameters

• Verify "Results field" specification works correctly
• Test "Overwrite" setting (true/false)
• Validate custom prompt text input
• Test Velocity code support in prompts (`$dotContentMap`)
• Verify asynchronous execution option
• Test model and temperature configuration

Functionality

• Test content generation from simple prompts
• Test content generation from complex prompts
• Verify `$dotContentMap` variable resolution
• Test JSON object handling in responses
• Validate field mapping of generated content
• Test async vs synchronous execution modes
• Verify overwrite vs preserve existing content behavior
• Test prompt with multiple Velocity variables

Edge Cases

• Test with invalid Velocity syntax in prompt
• Test with prompts referencing non-existent fields
• Test with very long prompts (token limits)
• Verify handling of malformed JSON responses
• Test with empty results field specification
• Test concurrent prompt executions

---

3. AI Embeddings Actionlet

Configuration & Parameters

• Test field list specification (comma-separated)
• Test field list specification (newline-separated)
• Verify index name reference configuration
• Test "Delete" operation mode
• Test "Insert" operation mode

Field Auto-Detection

• Verify Block Editor field detection for Contentlets
• Verify WYSIWYG field detection for Contentlets
• Verify TextArea field detection for Contentlets
• Test rendered HTML detection for Pages
• Test first binary field detection for File Assets
• Test first binary field detection for dotAssets
• Validate auto-detection accuracy across content types

Functionality

• Test embedding creation for specified fields
• Test embedding deletion for specified fields
• Verify index operations complete successfully
• Test embedding accuracy and search relevance
• Validate operation reversibility (insert then delete)
• Test with multiple fields in single operation
• Verify index persistence across content updates

Edge Cases

• Test with non-existent index name
• Test with invalid field names
• Test with empty field list
• Test with fields containing no text content
• Verify handling of binary field format variations
• Test concurrent embedding operations on same content

---

4. AI Image Auto-Tagging and Descriptions Actionlet

Configuration

• Verify `AI_VISION_MODEL` configuration (default: `gpt-4o`)
• Test `AI_VISION_MAX_TOKENS` configuration (default: `500`)
• Validate custom vision model selection
• Test token limit enforcement

Field Variables

• Verify `dotAITagSrc` variable detection on tag fields
• Verify `dotAIDescriptionSrc` variable detection on text fields
• Test field variable references to image field variable
• Validate proper field mapping

Triggering Mechanisms

• Test manual workflow action inclusion
• Test automatic triggering on Publish when field variables defined
• Verify publish-time execution timing
• Test with different publish workflow configurations

Functionality

• Test alt text generation from various image types (JPEG, PNG, SVG)
• Test metadata tag generation from image content
• Verify vision model accuracy for different image subjects
• Test with high-resolution images
• Test with low-resolution images
• Validate generated alt text quality and relevance
• Verify generated tags accuracy

Firing Inhibitors

• Verify existing text in target field prevents alt text generation
• Test `dot:taggedbydotai` tag presence prevents re-tagging
• Validate inhibitor logic with partial data
• Test forced re-processing after clearing inhibitors

Edge Cases

• Test with corrupted image files
• Test with unsupported image formats
• Test with very large images (size limits)
• Test with images containing no recognizable content
• Verify handling when image field is empty
• Test with multiple image fields in same content

---

5. AI Translate Content Actionlet

Configuration & Parameters

• Test translation target languages (comma-separated locale codes)
• Verify field types to translate (Text, WYSIWYG, Textarea, Block Editor)
• Test specific field variables inclusion
• Test specific field variables exclusion
• Validate language variable prefix for glossary lookup
• Verify `AI_TRANSLATION_SYSTEM_PROMPT` configuration
• Verify `AI_TRANSLATION_USER_PROMPT` configuration
• Test `AI_TRANSLATION_MODEL_KEY` (default: `gpt-4o`)
• Validate `AI_TRANSLATIONS_MAX_TOKENS` configuration
• Test `AI_TRANSLATION_TEMPERATURE` (default: `0.1`)
• Verify `AI_TRANSLATION_RESPONSE_FORMAT` (default: `json_format`)

Template Variables

• Verify `${sourceLanguage}` variable substitution
• Verify `${targetLanguage}` variable substitution
• Verify `${sourceJson}` variable substitution
• Verify `${translationKeys}` variable substitution

Functionality

• Test translation to single target language
• Test translation to multiple target languages simultaneously
• Verify HTML tag preservation in translations
• Verify JSON tag preservation in translations
• Test nested JSON structure handling
• Validate sequential translation accuracy
• Test glossary application when configured
• Verify translation quality and context awareness
• Test with different field types (Text, WYSIWYG, Textarea, Block Editor)

Multi-Language Content

• Test with content in various source languages
• Verify translation consistency across languages
• Test with right-to-left languages (Arabic, Hebrew)
• Test with special character languages (Chinese, Japanese, Korean)
• Validate locale code handling
• Test with non-standard language variants

Edge Cases

• Test with empty content fields
• Test with very long content (token limits)
• Test with content containing HTML entities
• Test with content containing special characters
• Test with malformed HTML in WYSIWYG fields
• Verify handling of unsupported target languages
• Test translation of content with embedded code blocks
• Test with glossary containing conflicting terms

---

Cross-Actionlet Testing

Asynchronous Execution

• Verify async execution warning message appears
• Test "save and exit" requirement for async operations
• Validate async timing across all actionlets
• Test user exit before async completion
• Verify incomplete saves are not lost during async operations
• Test async state management and recovery

Workflow Integration

• Test each actionlet in different workflow schemes
• Verify actionlet ordering when multiple AI actions present
• Test with conditional workflow executions
• Validate actionlet execution in sub-actions
• Test with workflow escalations and timeouts
• Verify proper workflow state transitions

Permission & Security

• Verify permission inheritance for generated content
• Test with different user roles and permissions
• Validate API key security and handling
• Test with expired or invalid API keys
• Verify audit logging for AI operations

Performance & Reliability

• Test with high content volume
• Verify response times within acceptable limits
• Test concurrent execution of multiple actionlets
• Validate system resource usage
• Test error recovery and retry mechanisms
• Verify timeout handling for long operations

Error Handling

• Test network connectivity issues during AI calls
• Verify proper error messages for all failure scenarios
• Test OpenAI API rate limiting handling
• Validate rollback behavior on failures
• Test partial completion handling
• Verify user notification of errors

---

Priority

High

---

Additional Context

• Test Against: Custom version built from branch `issue-32639-ai-dotai-portlet`
• Branch URL: https://github.com/dotCMS/core/tree/issue-32639-ai-dotai-portlet
• Prerequisite: dotAI must be configured with valid OpenAI API keys
• Critical Warning: Async functions require users to save and exit editor before background processing
• Related Documentation: https://dev.dotcms.com/docs/dotai-workflows
• Related Issues:
  • #34072 (dotAI Tool regression testing)
  • #34075 (Block Editor AI regression testing)
• Test Environment: Should be tested in both dev and staging with proper workflow configurations
• Test Data: Requires diverse content including various languages, image types, and field configurations