FasterXML / FasterXML/jackson-dataformats-binary

Invalid Avro file produced using SequenceWriter

未关闭
#339 7 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
avro
主要语言
Java
星标
347
派生
156
平均合并
3 天 3 小时
30 天内合并 PR
22

描述

While documentation on writing Avro to a file is sparse, I have managed to piece some stuff together but I am still getting an error.

- https://groups.google.com/g/jackson-user/c/yOXaJvAMzfg
- https://github.com/FasterXML/jackson-dataformats-binary/issues/35
- https://github.com/FasterXML/jackson-dataformats-binary/issues/15

Here is some sample code:

```java
final var avroFactory = AvroFactory.builderWithApacheDecoder().enable(AvroGenerator.Feature.AVRO_FILE_OUTPUT).build();

final var generator = new AvroSchemaGenerator().enableLogicalTypes();

final var mapper = AvroMapper.builder(avroFactory).addModule(new AvroJavaTimeModule()).build();
mapper.acceptJsonFormatVisitor(Thing.class, generator);

final var avroSchema = generator.getGeneratedSchema();

final var file = Files.createTempFile("something", ".avro").toFile();

final var out = new ByteArrayOutputStream();
final var writer = mapper.writer(avroSchema).writeValues(out);

// in a loop
writer.write(thing);

// after loop
writer.close();

try (FileOutputStream outputStream = new FileOutputStream(file)) {
out.writeTo(outputStream);
}
```

When checking the resultant file using [`avro-tools`](https://avro.apache.org/), I get the following error:

```
avro-tools tojson something.avro

22/09/08 18:36:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" org.apache.avro.AvroRuntimeException: java.io.IOException: Invalid sync!
at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:224)
at org.apache.avro.tool.DataFileReadTool.run(DataFileReadTool.java:97)
at org.apache.avro.tool.Main.run(Main.java:67)
at org.apache.avro.tool.Main.main(Main.java:56)
Caused by: java.io.IOException: Invalid sync!
at org.apache.avro.file.DataFileStream.nextRawBlock(DataFileStream.java:319)
at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:213)
... 3 mor
```

According to some searching, the `Invalid sync!` error occurs when the file hasn't been stitched together properly, but it's unclear to me what I need to do in code to get that to happen. I've looked through most of the Avro tests in this repo and I cannot find one that actually writes to a file and then de-serializes from that file.

I am not sure if I have stumbled into an actual bug here or not, but I am happy to try and write a test case if this code does seem correct since that would imply it's a bug?

Thanks in advance.

Edit:

I've also tried the following:

```java
final var file = Files.createTempFile("something", ".avro").toFile();
final SequenceWriter writer = mapper.writer(avroSchema).writeValues(file);
```

In which case I get the following error at that line:

```
java.lang.UnsupportedOperationException: Generator of type com.fasterxml.jackson.core.json.UTF8JsonGenerator does not support schema of type 'avro'

at com.fasterxml.jackson.core.JsonGenerator.setSchema(JsonGenerator.java:592)
at com.fasterxml.jackson.databind.ObjectWriter$GeneratorSettings.initialize(ObjectWriter.java:1393)
at com.fasterxml.jackson.databind.ObjectWriter._configureGenerator(ObjectWriter.java:1258)
at com.fasterxml.jackson.databind.ObjectWriter.createGenerator(ObjectWriter.java:717)
at com.fasterxml.jackson.databind.ObjectWriter.writeValues(ObjectWriter.java:753)
```

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。