Azure / Azure/azure-search-vector-samples
New AI search takes more storage space than earlier Azure cognitive search version
- Dominant language
- Jupyter Notebook
- Stars
- 910
- Forks
- 377
- PR merge metrics
- No merged PRs in 30d
Description
We use Azure cognitive search in one of our applications as a read model, I december 2024 we thought of upgrading ourselves from traditional Azure cognitive search to now named a Azure AI search and created a new basic tier service.
Right now in Old azure coginitve search we have approx 5M of documents which takes around 6GB of total storage as shown in image below. Below are the specifications of the same:
1. Created in 2021 tentative
2. Tier : Standard
3. Replicas : 3
4. Partitions: 1
5. Total Documents: ~5M
6. Storage occupied: 6GB

Upon moving to new AI search we rebuilt the indexes with the same ARM template and function that is responsible for creating documents for old one as shown in image below. Below are the observations after creation
1. Created in December end 2024
2. Tier: Basic
3. Replicas: 3
4. Partitions: 1
5. Total Documents: ~5M(Same documents as in old version with some additional documents as its in use now)
6. Storage: ~11GB

Question:
1. Why is there so much of a difference between these two in terms of storage?
2. What`s new in AI search that needs more space?
3. What can be done to reduce the space issue?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by comparing the ARM template and document-creation function used for the old and new Azure AI Search indexes, then inspect their index definitions, service tiers, and storage metrics. Confirm whether the documents and fields are equivalent, identify what accounts for the storage difference, and document a supported way to reduce storage or explain why it is expected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure
- Domain
- cloud, search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100