apache / apache/beam

Inputs SQS with Session based Windowing doesn't work

Open
#19,406 0 comments 0 reactions 0 assignees View on GitHub
aws bug io java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Hi,

 

Trying to use Beam with AWS SQS service as an input source, using Session windows.

The windows aren't executed. Code works well when the input source is Kafka:
```

// code placeholder
SimpleDateFormat format = new SimpleDateFormat("yyyy-MM-dd HH:mm");

PipelineOptions
options = PipelineOptionsFactory.create();
AwsOptions awsOptions = options.as(AwsOptions.class);
BasicAWSCredentials
awsCreds = new BasicAWSCredentials("", "");
awsOptions.setAwsCredentialsProvider(new AWSStaticCredentialsProvider(awsCreds));
awsOptions.setAwsRegion("eu-west-1");

Pipeline
p = Pipeline.create(options);
// This example reads a public data set consisting of the complete works
of Shakespeare.
p.apply(SqsIO.read().withQueueUrl("https://sqs.eu-west-1.amazonaws.com/XXXXXXXXXXXXXXXX"))

/*Per
session windows*/
.apply(ParDo.of(new DoFn() {
@ProcessElement
public void processElement(@Element
Message element, OutputReceiver out) {
// Extract the timestamp from log entry we're currently
processing.

LOG.info("Message Body: {}", element.getBody());
out.output(element.getBody());
}
}))

//Set
windowing configuration

.apply(
"WindowIntoSessions",
Window.into(
Sessions.withGapDuration(Duration.standardSeconds(5)))
.withTimestampCombiner(TimestampCombiner.END_OF_WINDOW)
//.triggering(Repeatedly.forever(AfterWatermark.pastEndOfWindow()))
//
Late data is dropped
.accumulatingFiredPanes()
.withAllowedLateness(Duration.ZERO))

//Extract and
count: Extracts a the object to an KV store of
.apply(
MapElements.into( TypeDescriptors.kvs(TypeDescriptors.strings(),
TypeDescriptors.integers()))
.via(
(String testO) -> KV.of(testO, new Integer(1))
)
)

.apply("CountElements",
Sum.integersPerKey())

.apply("Log", ParDo.of(new FilterTextFn()))

.apply(
MapElements.into(TypeDescriptors.strings())
.via(
(KV wordCount) ->
wordCount.getKey() + ": " + wordCount.getValue()))
;

p.run().waitUntilFinish();
}

```

 

Imported from Jira [BEAM-7498](https://issues.apache.org/jira/browse/BEAM-7498). Original Jira may contain additional context.
Reported by: esteveavi.

Contributor guide

Open the contributing guide

Research direction

Start with the SqsIO input path and its interaction with Beam session windows, using the supplied Kafka comparison as the behavioral reference. Check how message timestamps and watermarks are handled before Window.into(Sessions...), then reproduce the example and verify that session windows execute and produce the expected counts.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.