Scaling SimpleChat

This feature request outlines strategies for scaling the SimpleChat application beyond a single instance to handle increased load and improve availability.

Core Scaling Strategy: Horizontal Scaling with Azure App Gateway

The primary method for scaling SimpleChat involves running multiple independent instances of the web application behind an Azure App Gateway. This allows for true horizontal scaling.

Concept:

Instead of relying on scaling up (increasing resources of a single instance) or scaling out within a single App Service Plan (which can have limitations), you deploy multiple, separate SimpleChat web app instances. These instances share the same backend configuration (database, identity) and traffic is distributed among them by an App Gateway.

Implementation Steps:

1. Deploy Additional Instances: Spin up one or more new Azure Web App instances for SimpleChat.
2. Share Configuration via Cosmos DB:
   • Configure the environment variables of the new instances to point to the same existing Azure Cosmos DB used by your original SimpleChat instance.
   • Key Point: SimpleChat loads its application settings directly from this shared Cosmos DB. Therefore, you do not need a separate mechanism (like downloading/uploading JSON settings) to configure new instances for this scaling scenario. They will automatically adopt the settings from the database.
3. Use Same App Registration: Ensure all SimpleChat instances (original and new) are configured to use the exact same Azure AD App Registration for authentication and authorization.
4. Deploy Azure App Gateway: Set up an Azure App Gateway in front of all your SimpleChat web app instances.
5. Configure Backend Pool: Add all the SimpleChat web app instances to the backend pool of the App Gateway.
6. Enable Session Affinity: This is crucial. Configure the App Gateway's HTTP listener settings to enable Cookie-based Session Affinity. This ensures that once a user starts a session with a particular backend instance, subsequent requests from that user during the same session are routed to the same instance. This maintains session state consistency.

Benefits:

• True Horizontal Scalability: Easily add or remove instances based on demand.
• Increased Availability: If one instance fails, the App Gateway can route traffic to healthy instances.
• Potential for Geo-Distribution: Combined with a globally distributed Cosmos DB, this architecture can support deploying SimpleChat instances across multiple data centers or regions.

Scaling the Search Component: Multiple AI Search Indexes

SimpleChat relies on Azure AI Search for indexing and querying uploaded files. A single AI Search index has limitations (e.g., potentially around 1.2 million files per index, though this limit can change).

Strategy:

To overcome the limitations of a single index, SimpleChat can be enhanced (or may already support) using multiple Azure AI Search services or indexes.

How it Works:

1. Configure SimpleChat to be aware of multiple AI Search endpoints.
2. When a user performs a search, SimpleChat queries all configured indexes simultaneously.
3. Results from all indexes are gathered.
4. SimpleChat re-ranks and scores the combined results based on relevance.
5. The top N results are returned to the user.

Benefits:

• Overcomes Single Index Limits: Scale search capacity far beyond the limits of one index by distributing files across multiple indexes.
• Increased Search Throughput: Potentially parallelize search operations across multiple AI Search services.

Summary

• Scale the SimpleChat web application horizontally by deploying multiple instances behind an Azure App Gateway with Session Affinity enabled.
• Configure new instances by pointing them to the same shared Cosmos DB and using the same App Registration.
• Scale the search capacity by configuring SimpleChat to use multiple Azure AI Search indexes/services.