How to clean up invalid BinarySchema
- Dominant language
- Java
- Stars
- 5.1k
- Forks
- 1.9k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 46
Description
I created a cache like this IgniteCache. In order to do upsert, I used EntryProcessor as follows.
EntryProcessor
```java
public class UpsertEntryProcessorV1
implements EntryProcessor {
@IgniteInstanceResource
private Ignite ignite;
@SuppressWarnings("unchecked")
@Override
public Void process(MutableEntry entry,
Object... arguments) throws EntryProcessorException {
if (entry == null || entry.getKey() == null) {
return null;
}
Map> params =
(Map>) arguments[0];
Map param = params.get(entry.getKey());
if (param == null || param.isEmpty()) {
return null;
}
BinaryObjectBuilder builder;
BinaryObject entryVal = entry.getValue();
if (entryVal != null) {
builder = entryVal.toBuilder();
} else {
String valueType = (String) arguments[1];
builder = ignite.binary().builder(valueType);
}
for (Map.Entry item : param.entrySet()) {
if (item.getKey() == null) {
continue;
}
builder.setField(item.getKey(), item.getValue());
}
entry.setValue(builder.build());
return null;
}
}
```
In this cache, I wrote 20,000 records, each record has 1 to 1,000 fields. I found that the ignite service has a lot of full gc, and even crashes directly. Through heap dump analysis, I found that BinarySchema occupies a lot of memory.
heap

**Eventually I figured out that the problem was when I was randomly writing to some fields of a record every time.**
when I write a record for the first time, a BinarySchema will be created.
The next time I update this record and write one more field, a new BinarySchema will be created and written to ./work/db/ binary_meta/, the old schema will not be cleared.
In the end, tens of thousands of BinarySchemas will be created, but there are only dozens of BinarySchemas that are actually valid in serialized storage.
The following are some places I found to store BinarySchema, is there a way to clean up these BinarySchema?
1. BinaryContext#descByCls#schemaReg
2. BinaryContext#schemas
3. CacheObjectBinaryProcessorImpl#metadataLocCache#metadata#schemas
Contributor guide
Research direction
Start by reproducing the random-field update pattern described in the issue and inspect the heap and ./work/db/binary_meta/ contents. Trace BinaryContext#descByCls#schemaReg, BinaryContext#schemas, and CacheObjectBinaryProcessorImpl#metadataLocCache#metadata#schemas to understand how schemas accumulate. Done means establishing a safe cleanup or lifecycle for obsolete BinarySchema instances while retaining schemas needed by serialized storage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100