Aiven-Open / Aiven-Open/karapace

Corrupt schema, different schemas with the same id found

未关闭
#876 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
634
派生
110
平均合并
4 天 7 小时
30 天内合并 PR
4

描述

# What happened?

We are experiencing incompatible schemas being registered under the same schema id quite frequently

# What did you expect to happen?

incompatible schemas should have different ids or fail to register if for the same subject (not the case, these are different subjects)

# What else do we need to know?

Version of karapace: `3.10.4`

We are experiencing quite frequent issues of corrupted schemas in our deployments that we haven't been able to pinpoint yet to a root cause. The issue is that at some point a non compatible schema gets assigned an already present schema id, thus breaking the deserialization of the already existent records using that schema. For instance we have a record schema with id 11 and then after a few months a message with schema `string` gets inserted as schema 11, making the messages with the previous schema unrecoverable unless doing some manual intervention.

Today I tried to dig deeper in the issue and try to recover the schemas by deleting the subject that corrupted the schema.

So I accessed the topic `_schemas`, listing all the schemas with schema id `11` (that is the one that got corrupted), names and values are redacted:

| key | value |
|---|---|
| {"keytype":"SCHEMA","subject":"corrupt-subject","version":1,"magic":1} | {"subject": "corrupt-subject","version": 1, "id": 11, "schema": "\"string\"", "deleted": false} |
| {"keytype":"SCHEMA","subject":"subject-1","version":1,"magic":1} | {"subject": "subject-1","version": 1,"id": 11,"schema": "","deleted": false} |
| {"keytype":"SCHEMA","subject":"subject-2","version":1,"magic":1} | {"subject": "subject-2","version": 1,"id": 11,"schema": "","deleted": false} |
| {"keytype":"SCHEMA","subject":"subject-3","version":1,"magic":1} | {"subject": "subject-3", "version": 1,"id": 11,"schema": "","deleted": false} |
|---|---|

Then we deleted the corrupt entry with:

```
❯ curl -X DELETE 'http://karapace-url/subjects/corrupt-subject'

```

And we got another entry in the `_schemas` topic, I presume to mark the subject as deleted:

| key | value |
|---|---|
| {"keytype":"DELETE_SUBJECT","subject":"corrupt-subject","magic":0} | {"subject": "corrupt-subject","version": 1}

So we checked the versions for schema 11 using the API and all looks good:
```
❯ curl 'http://karapace-url/schemas/ids/11/versions' | jq -r
[
{
"subject": "subject-1",
"version": 1
},
{
"subject": "subject-2",
"version": 1
},
{
"subject": "subject2",
"version": 1
}
]

❯ curl 'http://karapace-url/subjects/subject-1/versions/1'
{"id":11,"schema":"","subject":"subject-1","version":1}

❯ curl 'http://karapace-url/subjects/subject-2/versions/1'
{"id":11,"schema":"","subject":"subject-2","version":1}

❯ curl 'http://karapace-url/subjects/subject-3/versions/1'
{"id":11,"schema":"","subject":"subject-3","version":1}
```

But when I try to retrieve schema 11 I get the corrupt entry again:

```
curl 'http://karapace-url/schemas/ids/11' | jq -r
{
"schema": "\"string\""
}
```

And indeed if I try to obtain the subject that I deleted I get an error:

```
❯ curl 'http://karapace-url/subjects/corrupt-subject/versions/1'
{"error_code":40401,"message":"Subject 'corrupt-subject' not found."}
```

I am not sure if we are doing some forbidden workflow, any help how to remove the corrupt schema (I know that all the entries for `corrupt-subject` won't be deserializable anymore) would be appreciated and also how we can debug the creation of corrupt entries, that are happening very frequently in our setup.

Thanks!

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。