OpenRefine / OpenRefine/OpenRefine

Fetching URLs in a large dataset ends with an OutOfMemoryError with no feedback for the user

Open
#2,257 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

error handling fetch urls large project support Type: Bug
Dominant language
Java
Stars
12k
Forks
2.2k
Avg merge
1d 14h
Merged PRs (30d)
27

Description

Describe the bug
A fetching URL's operation on a large dataset runs out of memory and gets stuck on 99%, giving no feedback to the user.

To Reproduce
Steps to reproduce the behavior:
Run an "Add column by fetching URL's" operation on ca 30k rows.

Current Results
Once the indicator on top of the page gets to 98 or 99%, it gets stuck there forever, giving no feedback to the user that something went wrong. When I check the console, I see:

10:38:08.848 [        project_utilities] Saved project '2337537703446' (2331ms)
Exception in thread "Thread-8" com.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2205)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3953)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3957)
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4875)
	at com.google.refine.operations.column.ColumnAdditionByFetchingURLsOperation$ColumnAdditionByFetchingURLsProcess.cachedFetch(ColumnAdditionByFetchingURLsOperation.java:331)
	at com.google.refine.operations.column.ColumnAdditionByFetchingURLsOperation$ColumnAdditionByFetchingURLsProcess.run(ColumnAdditionByFetchingURLsOperation.java:292)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

Expected behavior
Either not run out of memory ;) or at least show an error message in the interface.

Desktop (please complete the following information):

  • OS: Ubuntu 18.04.3 LTS
  • Browser Version: Chrome Version 78.0.3904.87 (Official Build) (64-bit)
  • JRE or JDK Version:
    openjdk version "1.8.0_232"
    OpenJDK Runtime Environment (build 1.8.0_232-8u232-b09-0ubuntu1~18.04.1-b09)
    OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)

OpenRefine (please complete the following information):

  • Version 3.3 Beta

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ColumnAdditionByFetchingURLsOperation.java, especially cachedFetch and run, using the stack trace and the 30k-row reproduction as entry points. Investigate how the operation reports failures to the interface; done means the large fetch either avoids the OutOfMemoryError or presents an error instead of remaining stuck at 98–99%.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.