apache / apache/beam

Beam Example(s) exploring public document datasets

Open
#18,124 0 comments 0 reactions 0 assignees View on GitHub
examples java mentor P3 wish
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

In Beam, we have examples illustrating counting the occurrences of words and performing a basic TF-IDF analysis on the works of Shakespeare (or whatever you point it at). It would be even cooler to do these analyses, and more, on a much larger data set that is really the subject of current investigations.

In chatting with professors at the University of Washington, I've learned that scholars of many fields would really like to explore new and highly customized ways of processing the growing body of publicly-available scholarly documents, such as PubMed Central. Queries like "show me documents where chemical compounds X and Y were both used in the 'method' section"

So I propose a Google Summer of Code project wherein a student writes some large-scale Beam pipelines to perform analyses such as term frequency, bigram frequency, etc.

Skills required:
- Java or Python
- (nice to have) Working through the Beam getting started materials

Imported from Jira [BEAM-1439](https://issues.apache.org/jira/browse/BEAM-1439). Original Jira may contain additional context.
Reported by: kenn.

Contributor guide

Open the contributing guide

Research direction

Start with Beam's existing word-count and TF-IDF examples and work through the Beam getting started materials. Read the original Jira issue for additional context, then define a specific public document dataset, analyses, pipeline scope, and acceptance criteria before implementation; done should be a working large-scale example with reproducible results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.