[TASK] Regression Testing for dotAI ViewTool

Description

Complete regression testing of the dotAI ViewTool (`$ai`) which provides AI operations accessible through Velocity scripts. The ViewTool enables text generation, image creation, semantic search, embeddings, and completion operations within VTL templates.

Key Capabilities:

• Text and Image Generation
• Semantic Search Operations
• Embeddings and Token Counting
• AI Completions and Summarization

Reference Documentation: https://dev.dotcms.com/docs/dotai-viewtool

---

Acceptance Criteria

1. Text Generation - `$ai.generateText()`

Parameter Variations

• Test with simple string prompt: `$ai.generateText("prompt text")`
• Test with map parameter containing custom options
• Verify prompt string encoding and special characters
• Test with very long prompts (token limits)
• Test with empty string prompt
• Test with null/undefined parameter

Response Structure

• Verify response contains `id` field
• Verify response contains `model` field
• Verify `choices` array is present
• Verify `choices[0].message.content` contains generated text
• Verify `usage` object contains token counts
• Validate JSON structure matches documentation

Functionality

• Test text generation with various prompt types
• Verify generated content quality and relevance
• Test with different models if configurable
• Verify temperature/creativity settings (if available)
• Test concurrent text generation requests
• Validate response time is acceptable

Error Handling

• Test with invalid parameters
• Verify error handling for API failures
• Test timeout behavior
• Verify error messages are meaningful

---

2. Image Generation - `$ai.generateImage()`

Parameter Variations

• Test with simple string prompt: `$ai.generateImage("image description")`
• Test with map parameter containing options
• Test with different image size specifications
• Test with style/quality parameters (if available)
• Test with empty string prompt
• Test with null/undefined parameter

Response Structure

• Verify `url` field contains valid image URL
• Verify `revised_prompt` field is populated
• Verify `originalPrompt` matches input
• Verify `tempFileName` is present
• Verify `response` object structure
• Validate all fields are correctly typed

Functionality

• Test image generation with various descriptions
• Verify generated images match prompts
• Validate image URL accessibility
• Test image download and storage
• Verify temp file cleanup
• Test revised prompt accuracy
• Validate image quality and resolution

Error Handling

• Test with inappropriate/filtered prompts
• Verify error handling for generation failures
• Test with prompts exceeding limits
• Validate timeout behavior

---

3. Search Operations - `$ai.search`

3.1 Query Search - `$ai.search.query(query, index)`

Parameter Variations

• Test with string query and string index
• Test with map query parameter
• Test with map containing custom options
• Test query without index (default index)
• Test with various index names
• Test with empty query string
• Test with complex/multi-word queries

Response Structure

• Verify `timeToEmbeddings` field is present
• Verify `total` count is accurate
• Verify `query` field echoes input
• Verify `threshold` field is present
• Verify `dotCMSResults` array structure
• Verify each result contains distance score
• Verify each result contains extracted text
• Verify `offset`, `limit`, `count` fields

Functionality

• Test semantic search with various queries
• Verify search relevance and ranking
• Test distance score accuracy
• Validate extracted text excerpts
• Test pagination with offset/limit
• Verify threshold filtering works correctly
• Test with multiple indexes
• Validate search performance

Edge Cases

• Test with non-existent index
• Test with queries returning no results
• Test with queries matching all content
• Test with very long query strings
• Test with special characters in queries
• Test concurrent search requests

3.2 Related Content - `$ai.search.related(content, index)`

Parameter Variations

• Test with contentlet object
• Test with different content types
• Test with specified index
• Test without index (default behavior)
• Test with null/invalid content object

Response Structure

• Verify response structure matches query response
• Verify similarity metrics are present
• Validate matched results accuracy

Functionality

• Test finding related content for various contentlets
• Verify similarity algorithm accuracy
• Test with content from different content types
• Validate ranking by similarity
• Test with content having no relations
• Test with recently indexed content

---

4. Embeddings - `$ai.embeddings`

4.1 Count Tokens - `$ai.embeddings.countTokens(prompt)`

Functionality

• Test token counting with short strings
• Test token counting with long strings
• Verify token count accuracy
• Test with various languages
• Test with special characters
• Test with empty string
• Test with HTML/markup content
• Validate integer return type

Accuracy Validation

• Compare counts with known benchmarks
• Test with strings of known token counts
• Verify consistency across multiple calls

4.2 Generate Embeddings - `$ai.embeddings.generateEmbeddings(prompt)`

Functionality

• Test embedding generation for simple text
• Test embedding generation for complex text
• Verify return value is floating-point array
• Validate array dimensions
• Test embedding consistency for same input
• Test with various text lengths
• Test with different languages

Validation

• Verify embeddings are semantically meaningful
• Test similarity of embeddings for related content
• Validate vector dimensions match model specs

4.3 Index Statistics - `$ai.embeddings.getIndexCount()` / `.indexCount`

Functionality

• Test `getIndexCount()` method
• Test `.indexCount` property access
• Verify both return same results
• Test with multiple indexes

Response Structure

• Verify `tokensPerChunk` field
• Verify `tokenTotal` field
• Verify `fragments` field
• Verify `contents` field
• Validate all numeric values are correct
• Test map structure and access

Validation

• Verify counts match actual index contents
• Test after index updates
• Test with empty indexes
• Test with large indexes

---

5. Completions - `$ai.completions`

5.1 Configuration - `$ai.completions.getConfig()` / `.config`

Functionality

• Test `getConfig()` method
• Test `.config` property access
• Verify both return same configuration

Response Structure

• Verify map contains model configurations
• Verify role prompt templates are present
• Verify text prompt templates are present
• Test supported models list (e.g., `gpt-3.5-turbo-16k`)
• Validate variable substitution patterns

Validation

• Verify configuration reflects current settings
• Test after configuration changes
• Validate all required config keys present

5.2 Summarize - `$ai.completions.summarize(prompt, [index])`

Parameter Variations

• Test with string prompt only
• Test with prompt and index name
• Test with prompt and no index (default)
• Test with empty prompt
• Test with very long prompt

Response Structure

• Verify search results are included
• Verify `openAiResponse` object is present
• Verify `openAiResponse.model` field
• Verify `openAiResponse.choices` array
• Verify `openAiResponse.usage` statistics
• Validate combined response structure

Functionality

• Test summarization of various content
• Verify summary quality and relevance
• Test with different index sources
• Validate search integration works correctly
• Test summary length/conciseness
• Verify explanation is meaningful

5.3 Raw Completions - `$ai.completions.raw(prompt)`

Parameter Variations

• Test with map parameter
• Test with JSON object
• Test with JSON string
• Test with custom completion parameters
• Test with system/user message arrays
• Test with temperature settings
• Test with max_tokens settings

Response Structure

• Verify raw OpenAI chat completion structure
• Verify `id`, `object`, `created` fields
• Verify `model` field
• Verify `choices` array structure
• Verify `usage` object
• Validate finish_reason values

Functionality

• Test custom completion scenarios
• Verify parameter passing to OpenAI API
• Test with various message configurations
• Validate response processing
• Test advanced completion features

---

Cross-Method Testing

Velocity Integration

• Test all methods accessible via `$ai` in VTL
• Verify method chaining where applicable
• Test variable assignment of return values
• Verify proper null handling in VTL
• Test conditional logic with AI responses
• Validate loop iteration over results

Response Serialization

• Test JSON serialization of all responses
• Verify nested object access in VTL
• Test array iteration in VTL
• Validate data type preservation

Configuration & Settings

• Test with different model configurations
• Verify API key handling
• Test with custom prompt templates
• Validate configuration override behavior
• Test with multiple concurrent configurations

Performance

• Measure response times for all methods
• Test concurrent method calls
• Verify caching behavior (if applicable)
• Test with high load scenarios
• Validate resource usage
• Test timeout configurations

Error Handling & Recovery

• Test all methods with invalid parameters
• Verify graceful degradation on API failures
• Test with network connectivity issues
• Validate error message clarity
• Test retry mechanisms (if applicable)
• Verify proper exception handling in VTL

Security

• Verify API key security in responses
• Test with restricted user permissions
• Validate input sanitization
• Test for injection vulnerabilities
• Verify sensitive data handling

---

Priority

High

---

Additional Context

• Test Against: Custom version built from branch `issue-32639-ai-dotai-portlet`
• Branch URL: https://github.com/dotCMS/core/tree/issue-32639-ai-dotai-portlet
• Testing Context: All methods should be tested within Velocity templates (VTL)
• Prerequisite: dotAI must be configured with valid OpenAI API keys
• Models: Support includes `gpt-3.5-turbo-16k` and other configurable models
• Related Documentation: https://dev.dotcms.com/docs/dotai-viewtool
• Related Issues:
  • #34072 (dotAI Tool regression testing)
  • #34075 (Block Editor AI regression testing)
  • #34112 (dotAI Workflows regression testing)
• Test Environment: Should be tested in both dev and staging with proper VTL contexts
• Test Pages: Create test pages/templates that exercise all ViewTool methods

Use the issue-32639-ai-dotai-portlet branch and the dotAI ViewTool documentation as the starting references. Configure dotAI with valid OpenAI API keys, create VTL test pages, and exercise the listed ViewTool methods in dev and staging. Done means the acceptance criteria, including response structures, errors, integration, performance, and security checks, are covered.