CentreForDigitalHumanities / CentreForDigitalHumanities/Word2VecElastic

Intervals may exclude final years

Open
#11 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

If the total number of years is not a multiple of the shift size, some of the last years in the data will not be included in any model.

For example, the EU parliament corpus has date range of 1999-2026. If I set the interval size to 10 and the shift to 5, the intervals are:

- 1999-2009
- 2004-2014
- 2009-2019
- 2014-2024

So 2025-2026 never get included.

It is not essential that intervals are all the same length, so perhaps it would be preferable if the last interval here was extended to 2014-2026. See also #8

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the interval construction and the code that selects corpus years for each model; no specific files or tests are named in the issue. Reproduce the 1999-2026 example with interval size 10 and shift 5, then verify that the final interval includes 2025-2026 without breaking the existing intervals.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.