[TASK] Regression Testing for dotAI REST API

Description

Complete regression testing of the dotAI REST API which provides headless access to OpenAI operations. The API offers endpoints for text generation, image generation, semantic search, embeddings management, and AI completions.

Base Path: /api/v1/ai

Key Capabilities:

• Text and Image Generation
• Semantic Search Operations
• Embeddings Management
• AI Completions and Configuration

Reference Documentation: https://dev.dotcms.com/docs/dotai-api

---

Acceptance Criteria

1. Text Generation Endpoints

1.1 Text Generation (GET/POST) - /api/v1/ai/text/generate

HTTP Methods

• Test GET request with query parameters
• Test POST request with JSON body
• Verify both methods return identical results for same input
• Test with Content-Type: application/json header

Request Parameters

• Test with simple text prompt
• Test with prompt containing special characters
• Test prompt length boundary (1-4096 characters)
• Test with maximum length prompt (4096 chars)
• Test with prompt exceeding 4096 chars (expect error)
• Test with empty prompt (expect error)
• Test with model parameter
• Test with temperature parameter (0-2 range)
• Test temperature clamping (values >2 should clamp to 2)

Response Structure

• Verify id field format (chatcmpl-...)
• Verify object field equals "chat.completion"
• Verify created timestamp is valid
• Verify model field matches requested model
• Verify choices array structure
• Verify choices[0].message.role equals "assistant"
• Verify choices[0].message.content contains generated text
• Verify choices[0].finish_reason is present
• Verify usage object contains token counts
• Validate prompt_tokens, completion_tokens, total_tokens

Status Codes

• Verify 200 OK for successful requests
• Verify 400 Bad Request for invalid parameters
• Verify 401 Unauthorized without authentication
• Verify 500 Internal Server Error for API failures

1.2 Image Test - /api/v1/ai/image/test

• Test endpoint accessibility
• Verify test response structure
• Validate test functionality

---

2. Image Generation Endpoints

2.1 Image Generation (GET/POST) - /api/v1/ai/image/generate

HTTP Methods

• Test GET request with query parameters
• Test POST request with JSON body
• Verify both methods work correctly

Request Parameters (AIImageRequestDTO)

• Test with required prompt parameter
• Test numberOfImages parameter (default: 1)
• Test multiple image generation (numberOfImages > 1)
• Test size parameter options
  • Test "1024x1024" (default)
  • Test other supported sizes
• Test quality parameter
  • Test "standard" (default)
  • Test "hd" quality
• Test style parameter
  • Test "natural" (default)
  • Test "vivid" style

Response Structure

• Verify revised_prompt field is present
• Verify url field contains valid image URL
• Verify originalPrompt matches input
• Verify tempFileName is generated
• Verify response field is present
• Verify tempFile path is accessible
• Test image URL accessibility
• Validate generated image format and quality

Edge Cases

• Test with empty prompt (expect error)
• Test with very long prompt
• Test with inappropriate content prompt
• Test numberOfImages boundary values
• Test invalid size parameter
• Test invalid quality/style parameters

---

3. Search Endpoints

3.1 Semantic Search (GET/POST) - /api/v1/ai/search

HTTP Methods

• Test GET request with query parameters
• Test POST request with JSON body
• Verify consistent results across methods

Request Parameters (CompletionsForm subset)

• Test required prompt parameter (1-4096 chars)
• Test searchLimit parameter (1-1000, default: 50)
• Test searchOffset parameter for pagination
• Test threshold parameter (0-1, default: 0.25)
• Test operator parameter
  • Test "cosine" (default)
  • Test "distance"
  • Test "innerProduct"
• Test contentType filtering (comma-separated)
• Test indexName parameter (default: "default")
• Test language parameter (ID or code)

Response Structure

• Verify results array structure
• Verify each result contains identifier
• Verify each result contains title
• Verify each result contains contentType
• Verify each result contains score (similarity)
• Verify each result contains content (excerpt)
• Verify totalResults count is accurate
• Verify query echoes input
• Verify threshold value

Functionality

• Test semantic search with various queries
• Verify search relevance and ranking
• Test pagination with limit/offset combinations
• Test threshold filtering (0, 0.5, 1.0)
• Test different operators affect ranking
• Test contentType filtering
• Test language filtering
• Test with multiple indexes

Edge Cases

• Test with query returning no results
• Test with query matching all content
• Test offset beyond total results
• Test limit exceeding 1000 (expect clamping)
• Test threshold outside 0-1 range

3.2 Find Related Content (GET/POST) - /api/v1/ai/search/related

Request Parameters

• Test with contentlet identifier
• Test with indexName parameter
• Test with threshold parameter
• Test with searchLimit parameter

Functionality

• Test finding related content for various contentlets
• Verify similarity ranking
• Test with different content types
• Validate related content accuracy

Edge Cases

• Test with non-existent identifier
• Test with content having no relations
• Test with newly created content

3.3 Search Test - /api/v1/ai/search/test

• Test endpoint accessibility
• Verify test response structure
• Validate test functionality

---

4. Embeddings Endpoints

4.1 Create Embeddings (POST) - /api/v1/ai/embeddings

Request Parameters (EmbeddingsForm)

• Test required query parameter (1-4096 chars)
• Test limit parameter (1-1000, default: 1000)
• Test offset parameter for batch processing
• Test indexName parameter (default: "default")
• Test model parameter
• Test fields parameter (comma-separated)
• Test velocityTemplate for custom extraction

Functionality

• Test embedding creation for single content
• Test batch embedding creation
• Test with different content types
• Test with custom field selection
• Test with Velocity template extraction
• Verify embeddings are stored correctly
• Test with multiple indexes
• Validate embedding vector generation

Edge Cases

• Test with empty query (expect error)
• Test with query exceeding 4096 chars
• Test with invalid field names
• Test with malformed Velocity template
• Test concurrent embedding creation

4.2 Delete Embeddings (DELETE) - /api/v1/ai/embeddings

Request Parameters

• Test with specific contentlet identifiers
• Test with indexName parameter
• Test batch deletion

Functionality

• Test deleting embeddings for single content
• Test deleting embeddings for multiple content
• Verify embeddings are removed from index
• Test deletion from specific index
• Test deletion from multiple indexes

Edge Cases

• Test deleting non-existent embeddings
• Test deleting from non-existent index
• Test concurrent deletion operations

4.3 Count Embeddings (GET/POST) - /api/v1/ai/embeddings/count

HTTP Methods

• Test GET request
• Test POST request

Request Parameters

• Test without parameters (count all)
• Test with indexName parameter
• Test with contentType filter

Response

• Verify count accuracy
• Test count after creation
• Test count after deletion
• Validate count per index

4.4 Index Count (GET) - /api/v1/ai/embeddings/indexCount

Authorization

• Verify requires CMS Administrator role
• Verify 401 for unauthenticated users
• Verify 403 for non-admin users

Response Structure

• Verify tokensPerChunk field
• Verify tokenTotal field
• Verify fragments count
• Verify contents count
• Test response for each index

Functionality

• Test with single index
• Test with multiple indexes
• Verify counts match actual data
• Test after index modifications

4.5 Drop/Recreate Table (DELETE) - /api/v1/ai/embeddings/db

Authorization

• Verify requires CMS Administrator role
• Verify 401 for unauthenticated users
• Verify 403 for non-admin users

Functionality

• Test table drop operation
• Verify table recreation
• Test data persistence after recreation
• Validate table schema

Safety

• Verify destructive operation confirmation
• Test operation cannot be undone
• Verify backup/recovery procedures

4.6 Embeddings Test - /api/v1/ai/embeddings/test

• Test endpoint accessibility
• Verify test response structure
• Validate test functionality

---

5. Completions Endpoints

5.1 Generate Completions (POST) - /api/v1/ai/completions

Request Parameters (CompletionsForm - Full)

• Test required prompt parameter (1-4096 chars)
• Test searchLimit (1-1000, default: 50)
• Test searchOffset for pagination
• Test responseLengthTokens (minimum: 128)
• Test threshold (0-1, default: 0.25)
• Test temperature (0-2, default: config)
• Test model parameter
• Test operator (distance/cosine/innerProduct)
• Test stream parameter (boolean)
• Test contentType filtering
• Test indexName (default: "default")
• Test language parameter

Streaming Response

• Test with stream=true
• Verify Content-Type: application/octet-stream
• Test streaming data chunks
• Verify complete response assembly
• Test stream interruption handling

Non-Streaming Response

• Test with stream=false (default)
• Verify Content-Type: application/json
• Verify response contains search results
• Verify timeToEmbeddings field
• Verify dotCMSResults array
• Verify openAiResponse object
• Validate combined response structure

Functionality

• Test completions with various prompts
• Verify search integration works
• Test with different models
• Test temperature effects on responses
• Validate response length control
• Test with different operators
• Test content type filtering
• Test language-specific completions

Edge Cases

• Test with responseLengthTokens < 128 (expect clamping)
• Test with temperature > 2 (expect clamping to 2)
• Test with threshold outside 0-1 range
• Test with very long prompts
• Test concurrent completion requests

5.2 Raw Prompt (POST) - /api/v1/ai/completions/rawPrompt

Request Format

• Test with raw JSON prompt
• Test with custom message arrays
• Test with system/user message combinations
• Test with various parameter combinations

Functionality

• Test direct OpenAI API access
• Verify parameter passthrough
• Test custom completion scenarios
• Validate raw response format

Edge Cases

• Test with invalid JSON structure
• Test with unsupported parameters
• Test with malformed message arrays

5.3 Configuration (GET) - /api/v1/ai/completions/config

Response Structure

• Verify apiUrl field
• Verify apiKey field (should be masked)
• Verify model names
• Verify embeddingsModel field
• Verify availableModels array
• Verify role prompt templates
• Verify text prompt templates

Functionality

• Test configuration retrieval
• Verify sensitive data masking (API key)
• Test after configuration changes
• Validate all expected fields present

Security

• Verify API key is never exposed in full
• Test with different authentication levels
• Validate permission requirements

---

Cross-Endpoint Testing

Authentication & Authorization

• Test all endpoints without authentication (expect 401)
• Test all endpoints with valid authentication
• Test admin-only endpoints with non-admin user (expect 403)
• Test admin-only endpoints with admin user (expect success)
• Verify session token handling
• Test with expired authentication tokens

HTTP Standards Compliance

• Verify proper HTTP status codes for all scenarios
• Test Content-Type headers on requests
• Verify Content-Type headers on responses
• Test with Accept header variations
• Verify CORS headers (if applicable)
• Test HTTP method validation (405 for unsupported)

Parameter Validation

• Test boundary values for all numeric parameters
• Test string length limits
• Test invalid parameter types
• Test missing required parameters
• Test extra/unknown parameters (should be ignored)
• Verify parameter type coercion

Error Handling

• Verify 400 Bad Request for invalid input
• Verify 401 Unauthorized for auth failures
• Verify 403 Forbidden for permission failures
• Verify 404 Not Found for non-existent resources
• Verify 500 Internal Server Error for API failures
• Test error message clarity and usefulness
• Verify error response format consistency

Performance & Scalability

• Test response times for all endpoints
• Test with concurrent requests
• Test with large payloads
• Test with high request volume
• Verify timeout handling
• Test rate limiting (if applicable)
• Validate resource cleanup

Data Consistency

• Test create → read → update → delete flows
• Verify index synchronization
• Test transaction rollback scenarios
• Validate data persistence
• Test with simultaneous operations

Integration Testing

• Test workflows combining multiple endpoints
• Test embeddings creation → search → completions flow
• Verify API consistency with ViewTool behavior
• Test with different content types
• Validate cross-index operations

---

Priority

High

---

Additional Context

• Test Against: Custom version built from branch issue-32639-ai-dotai-portlet
• Branch URL: https://github.com/dotCMS/core/tree/issue-32639-ai-dotai-portlet
• Base Path: All endpoints under /api/v1/ai
• Authentication: All endpoints require user authentication
• Admin Endpoints: indexCount and db deletion require CMS Administrator role
• Testing Tools: Use REST clients (Postman, curl, etc.) and automated API testing frameworks
• Related Documentation: https://dev.dotcms.com/docs/dotai-api
• Related Issues:
  • #34072 (dotAI Tool regression testing)
  • #34075 (Block Editor AI regression testing)
  • #34112 (dotAI Workflows regression testing)
  • #34113 (dotAI ViewTool regression testing)
• Test Environment: Should be tested in both dev and staging with proper authentication
• Test Data: Requires diverse content, multiple indexes, and various content types