linkedin / linkedin/transport

Transport plugin shade all by default which cause type not found issue when refer to other libs

Open
#112 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
306
Forks
75
PR merge metrics
No merged PRs in 30d

Description

When I create a new UDF function which calls the API exposed by another lib, it errored when I tested in HIVE and other platforms.

```
Caused by: com_linkedin_jobs_udf_jobs_udfs_2_1_1.org.apache.avro.AvroTypeException: Found com.linkedin.standardization.taxonomy.industries.IndustryStatus, expecting com_linkedin_jobs_udf_jobs_udfs_2_1_1.com.linkedin.standardization.taxonomy.industries.IndustryStatus
at com_linkedin_jobs_udf_jobs_udfs_2_1_1.org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:309)
at com_linkedin_jobs_udf_jobs_udfs_2_1_1.org.apache.avro.io.parsing.Parser.advance(Parser.java:86)
at com_linkedin_jobs_udf_jobs_udfs_2_1_1.org.apache.avro.io.ResolvingDecoder.readEnum(ResolvingDecoder.java:260)
```
The reason is that it expects the shaded type prefixed with ```com_linkedin_jobs_udf_jobs_udfs_2_1_1``` but found the original type.

To workaround it, we have to explicitly exclude those namespaces by adding
```
shadeHiveJar.setDoNotShade(["com.linkedin.standardization.taxonomy.industries.*"])
shadeSpark_211Jar.setDoNotShade(["com.linkedin.standardization.taxonomy.industries.*"])
shadeSpark_212Jar.setDoNotShade(["com.linkedin.standardization.taxonomy.industries.*"])
```
to ```build.gradle```.

Is this by design or a bug?

For more information, please refer to the internal discussion: https://linkedin-randd.slack.com/archives/C02D9EYGPGA/p1641401436435300

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read build.gradle and the shadeHiveJar, shadeSpark_211Jar, and shadeSpark_212Jar configuration first. Reproduce the Avro type mismatch with a UDF that calls another library, then determine the intended default shading behavior. Done means the bug/design question has an agreed, verifiable outcome.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.