apache / apache/pinot

ADLS Gen2 fails to overwrite segment during segment upload for the existing segment

Open
#10,267 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

```
023/02/08 15:33:51.265 ERROR [LaunchDataIngestionJobCommand] [main] Got exception to kick off standalone data ingestion job -
java.lang.RuntimeException: Caught exception during running - org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:148) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30
c4635bfeee88f88aa9c9f63b93bcd4a650607f]
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:118) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c463
5bfeee88f88aa9c9f63b93bcd4a650607f]
at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:121) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10
.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
at org.apache.pinot.tools.Command.call(Command.java:33) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
at org.apache.pinot.tools.Command.call(Command.java:29) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
at picocli.CommandLine.executeUserObject(CommandLine.java:1953) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
at picocli.CommandLine.access$1300(CommandLine.java:145) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee8
8f88aa9c9f63b93bcd4a650607f]
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a6
50607f]
at picocli.CommandLine.execute(CommandLine.java:2078) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:161) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b9
3bcd4a650607f]
at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:192) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bc
d4a650607f]
Caused by: java.lang.RuntimeException: org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 1 attempts
at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner.run(SegmentTarPushJobRunner.java:90) ~[pinot-batch-ingestion-standalone-0.10.0-shaded.ja
r:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:146) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30
c4635bfeee88f88aa9c9f63b93bcd4a650607f]
... 13 more
Caused by: org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 1 attempts
at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:61) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93b
cd4a650607f]
at org.apache.pinot.segment.local.utils.SegmentPushUtils.pushSegments(SegmentPushUtils.java:123) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88
aa9c9f63b93bcd4a650607f]
at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner.run(SegmentTarPushJobRunner.java:88) ~[pinot-batch-ingestion-standalone-0.10.0-shaded.ja
r:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:146) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30
c4635bfeee88f88aa9c9f63b93bcd4a650607f]
... 13 more
java.lang.RuntimeException: Caught exception during running - org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:148)
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:118)
at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:121)
at org.apache.pinot.tools.Command.call(Command.java:33)
at org.apache.pinot.tools.Command.call(Command.java:29)
at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
at picocli.CommandLine.access$1300(CommandLine.java:145)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
at picocli.CommandLine.execute(CommandLine.java:2078)
at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:161)
at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:192)
Caused by: java.lang.RuntimeException: org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 1 attempts
at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:61)
at org.apache.pinot.segment.local.utils.SegmentPushUtils.pushSegments(SegmentPushUtils.java:123)
at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner.run(SegmentTarPushJobRunner.java:88)
... 14 more
```

Contributor guide

Open the contributing guide

Research direction

Start with SegmentPushUtils.pushSegments and SegmentTarPushJobRunner, the entry points named in the stack trace. Reproduce segment upload against ADLS Gen2 when the segment already exists, then verify that the existing segment is overwritten without the AttemptsExceededException shown here.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, java
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.