Azure / Azure/azure-search-vector-samples

New AI search takes more storage space than earlier Azure cognitive search version

Open
#302 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
910
Forks
377
PR merge metrics
No merged PRs in 30d

Description

We use Azure cognitive search in one of our applications as a read model, I december 2024 we thought of upgrading ourselves from traditional Azure cognitive search to now named a Azure AI search and created a new basic tier service.

Right now in Old azure coginitve search we have approx 5M of documents which takes around 6GB of total storage as shown in image below. Below are the specifications of the same:
1. Created in 2021 tentative
2. Tier : Standard
3. Replicas : 3
4. Partitions: 1
5. Total Documents: ~5M
6. Storage occupied: 6GB

![Image](https://github.com/user-attachments/assets/26849ab1-a0e4-432d-839e-c0f52f69dca9)

Upon moving to new AI search we rebuilt the indexes with the same ARM template and function that is responsible for creating documents for old one as shown in image below. Below are the observations after creation
1. Created in December end 2024
2. Tier: Basic
3. Replicas: 3
4. Partitions: 1
5. Total Documents: ~5M(Same documents as in old version with some additional documents as its in use now)
6. Storage: ~11GB

![Image](https://github.com/user-attachments/assets/a211d6cf-f3f3-4e7a-b464-83a4bbb3a346)

Question:
1. Why is there so much of a difference between these two in terms of storage?
2. What`s new in AI search that needs more space?
3. What can be done to reduce the space issue?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by comparing the ARM template and document-creation function used for the old and new Azure AI Search indexes, then inspect their index definitions, service tiers, and storage metrics. Confirm whether the documents and fields are equivalent, identify what accounts for the storage difference, and document a supported way to reduce storage or explain why it is expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
cloud, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.