linkedin / linkedin/avro-util

Unable to compile generated class when avro schema name is too long

Open
#579 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
87
Forks
73
Avg merge
2h 50m
Merged PRs (30d)
1

Description

When the name of an avro schema is too long, the Java class is unable to be created as the file name becomes too long. This occurs because the `SchemaAssistant` class returns the name of the schema when the type is a record, and uses that as a component of the generated class name in `FastDeserializerGeneratorBase` and `FastSerializerGenerator`.

An example stack trace for a failure in creating the Deserializer is
```
com.linkedin.avro.fastserde.FastDeserializerGeneratorException: java.io.FileNotFoundException: /tmp/generated18270964553371151670/com/linkedin/avro/fastserde/generated/deserialization/AVRO_1_11/StrategyKey_AND_FilteringData_JOINED_AND_PreviousDayPricingParametersDerivativePricer_v1_AND_PricingParametersDerivativePricer_v1_JOINED_AND_AutoPriceOffsetParameters_JOINED_AND_JarvisBidAskOffsetParameters_JOINED_AND_AutoPriceOffset_JOINED_AND_ManualPriceOffset_JOINED_AND_VolatilityOffset_JOINED_JOINED_AND_AutoBidAskOffset_AND_DimerParameters_JOINED_AND_QuoterParameters_JOINED_AND_HitterParameters_JOINED_AND_InstrumentTradingParameters_JOINED_AND_ListingPricingParameters_JOINED_JOINED_GenericDeserializer_385168959_385168959.java (File name too long)
at com.linkedin.avro.fastserde.FastDeserializerGenerator.generateDeserializer(FastDeserializerGenerator.java:181)
at com.linkedin.avro.fastserde.FastSerdeCache.buildFastGenericDeserializer(FastSerdeCache.java:503)
at com.linkedin.avro.fastserde.FastSerdeCache.buildGenericDeserializer(FastSerdeCache.java:531)
at com.linkedin.avro.fastserde.FastSerdeCache.lambda$getFastGenericDeserializer$4(FastSerdeCache.java:276)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.io.FileNotFoundException: /tmp/generated18270964553371151670/com/linkedin/avro/fastserde/generated/deserialization/AVRO_1_11/StrategyKey_AND_FilteringData_JOINED_AND_PreviousDayPricingParametersDerivativePricer_v1_AND_PricingParametersDerivativePricer_v1_JOINED_AND_AutoPriceOffsetParameters_JOINED_AND_JarvisBidAskOffsetParameters_JOINED_AND_AutoPriceOffset_JOINED_AND_ManualPriceOffset_JOINED_AND_VolatilityOffset_JOINED_JOINED_AND_AutoBidAskOffset_AND_DimerParameters_JOINED_AND_QuoterParameters_JOINED_AND_HitterParameters_JOINED_AND_InstrumentTradingParameters_JOINED_AND_ListingPricingParameters_JOINED_JOINED_GenericDeserializer_385168959_385168959.java (File name too long)
at java.base/java.io.FileOutputStream.open0(Native Method)
at java.base/java.io.FileOutputStream.open(FileOutputStream.java:293)
at java.base/java.io.FileOutputStream.(FileOutputStream.java:235)
at java.base/java.io.FileOutputStream.(FileOutputStream.java:184)
at com.sun.codemodel.writer.FileCodeWriter.openBinary(FileCodeWriter.java:91)
at com.sun.codemodel.CodeWriter.openSource(CodeWriter.java:100)
at com.sun.codemodel.writer.FilterCodeWriter.openSource(FilterCodeWriter.java:67)
at com.sun.codemodel.writer.ProgressCodeWriter.openSource(ProgressCodeWriter.java:76)
at com.sun.codemodel.JPackage.createJavaSourceFileWriter(JPackage.java:486)
at com.sun.codemodel.JPackage.build(JPackage.java:437)
at com.sun.codemodel.JCodeModel.build(JCodeModel.java:311)
at com.sun.codemodel.JCodeModel.build(JCodeModel.java:280)
at com.sun.codemodel.JCodeModel.build(JCodeModel.java:259)
at com.sun.codemodel.JCodeModel.build(JCodeModel.java:287)
at com.linkedin.avro.fastserde.FastSerdeBase.compileClass(FastSerdeBase.java:165)
at com.linkedin.avro.fastserde.FastDeserializerGenerator.generateDeserializer(FastDeserializerGenerator.java:173
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with SchemaAssistant and trace how record names flow into FastDeserializerGeneratorBase and FastSerializerGenerator, using the stack trace as the entry point. Inspect the generated class-name construction and compilation path in FastDeserializerGenerator and FastSerdeBase. Done means a schema with a very long name can generate and compile without a FileNotFoundException caused by the output filename.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.