Transport plugin shade all by default which cause type not found issue when refer to other libs
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 306
- Forks
- 75
- PR merge metrics
- No merged PRs in 30d
Description
When I create a new UDF function which calls the API exposed by another lib, it errored when I tested in HIVE and other platforms.
```
Caused by: com_linkedin_jobs_udf_jobs_udfs_2_1_1.org.apache.avro.AvroTypeException: Found com.linkedin.standardization.taxonomy.industries.IndustryStatus, expecting com_linkedin_jobs_udf_jobs_udfs_2_1_1.com.linkedin.standardization.taxonomy.industries.IndustryStatus
at com_linkedin_jobs_udf_jobs_udfs_2_1_1.org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:309)
at com_linkedin_jobs_udf_jobs_udfs_2_1_1.org.apache.avro.io.parsing.Parser.advance(Parser.java:86)
at com_linkedin_jobs_udf_jobs_udfs_2_1_1.org.apache.avro.io.ResolvingDecoder.readEnum(ResolvingDecoder.java:260)
```
The reason is that it expects the shaded type prefixed with ```com_linkedin_jobs_udf_jobs_udfs_2_1_1``` but found the original type.
To workaround it, we have to explicitly exclude those namespaces by adding
```
shadeHiveJar.setDoNotShade(["com.linkedin.standardization.taxonomy.industries.*"])
shadeSpark_211Jar.setDoNotShade(["com.linkedin.standardization.taxonomy.industries.*"])
shadeSpark_212Jar.setDoNotShade(["com.linkedin.standardization.taxonomy.industries.*"])
```
to ```build.gradle```.
Is this by design or a bug?
For more information, please refer to the internal discussion: https://linkedin-randd.slack.com/archives/C02D9EYGPGA/p1641401436435300
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read build.gradle and the shadeHiveJar, shadeSpark_211Jar, and shadeSpark_212Jar configuration first. Reproduce the Avro type mismatch with a UDF that calls another library, then determine the intended default shading behavior. Done means the bug/design question has an agreed, verifiable outcome.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100