Aiven-Open / Aiven-Open/karapace
Corrupt schema, different schemas with the same id found
- 主要语言
- Python
- 星标
- 634
- 派生
- 110
- 平均合并
- 4 天 7 小时
- 30 天内合并 PR
- 4
描述
# What happened?
We are experiencing incompatible schemas being registered under the same schema id quite frequently
# What did you expect to happen?
incompatible schemas should have different ids or fail to register if for the same subject (not the case, these are different subjects)
# What else do we need to know?
Version of karapace: `3.10.4`
We are experiencing quite frequent issues of corrupted schemas in our deployments that we haven't been able to pinpoint yet to a root cause. The issue is that at some point a non compatible schema gets assigned an already present schema id, thus breaking the deserialization of the already existent records using that schema. For instance we have a record schema with id 11 and then after a few months a message with schema `string` gets inserted as schema 11, making the messages with the previous schema unrecoverable unless doing some manual intervention.
Today I tried to dig deeper in the issue and try to recover the schemas by deleting the subject that corrupted the schema.
So I accessed the topic `_schemas`, listing all the schemas with schema id `11` (that is the one that got corrupted), names and values are redacted:
| key | value |
|---|---|
| {"keytype":"SCHEMA","subject":"corrupt-subject","version":1,"magic":1} | {"subject": "corrupt-subject","version": 1, "id": 11, "schema": "\"string\"", "deleted": false} |
| {"keytype":"SCHEMA","subject":"subject-1","version":1,"magic":1} | {"subject": "subject-1","version": 1,"id": 11,"schema": "","deleted": false} |
| {"keytype":"SCHEMA","subject":"subject-2","version":1,"magic":1} | {"subject": "subject-2","version": 1,"id": 11,"schema": "","deleted": false} |
| {"keytype":"SCHEMA","subject":"subject-3","version":1,"magic":1} | {"subject": "subject-3", "version": 1,"id": 11,"schema": "","deleted": false} |
|---|---|
Then we deleted the corrupt entry with:
```
❯ curl -X DELETE 'http://karapace-url/subjects/corrupt-subject'
```
And we got another entry in the `_schemas` topic, I presume to mark the subject as deleted:
| key | value |
|---|---|
| {"keytype":"DELETE_SUBJECT","subject":"corrupt-subject","magic":0} | {"subject": "corrupt-subject","version": 1}
So we checked the versions for schema 11 using the API and all looks good:
```
❯ curl 'http://karapace-url/schemas/ids/11/versions' | jq -r
[
{
"subject": "subject-1",
"version": 1
},
{
"subject": "subject-2",
"version": 1
},
{
"subject": "subject2",
"version": 1
}
]
❯ curl 'http://karapace-url/subjects/subject-1/versions/1'
{"id":11,"schema":"","subject":"subject-1","version":1}
❯ curl 'http://karapace-url/subjects/subject-2/versions/1'
{"id":11,"schema":"","subject":"subject-2","version":1}
❯ curl 'http://karapace-url/subjects/subject-3/versions/1'
{"id":11,"schema":"","subject":"subject-3","version":1}
```
But when I try to retrieve schema 11 I get the corrupt entry again:
```
curl 'http://karapace-url/schemas/ids/11' | jq -r
{
"schema": "\"string\""
}
```
And indeed if I try to obtain the subject that I deleted I get an error:
```
❯ curl 'http://karapace-url/subjects/corrupt-subject/versions/1'
{"error_code":40401,"message":"Subject 'corrupt-subject' not found."}
```
I am not sure if we are doing some forbidden workflow, any help how to remove the corrupt schema (I know that all the entries for `corrupt-subject` won't be deserializable anymore) would be appreciated and also how we can debug the creation of corrupt entries, that are happening very frequently in our setup.
Thanks!
贡献指南
评估
这个 Issue 还没有评估数据。