OpenRefine / OpenRefine/OpenRefine
Fetching URLs in a large dataset ends with an OutOfMemoryError with no feedback for the user
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 12k
- Forks
- 2.2k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 27
Description
Describe the bug
A fetching URL's operation on a large dataset runs out of memory and gets stuck on 99%, giving no feedback to the user.
To Reproduce
Steps to reproduce the behavior:
Run an "Add column by fetching URL's" operation on ca 30k rows.
Current Results
Once the indicator on top of the page gets to 98 or 99%, it gets stuck there forever, giving no feedback to the user that something went wrong. When I check the console, I see:
10:38:08.848 [ project_utilities] Saved project '2337537703446' (2331ms)
Exception in thread "Thread-8" com.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: GC overhead limit exceeded
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2205)
at com.google.common.cache.LocalCache.get(LocalCache.java:3953)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3957)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4875)
at com.google.refine.operations.column.ColumnAdditionByFetchingURLsOperation$ColumnAdditionByFetchingURLsProcess.cachedFetch(ColumnAdditionByFetchingURLsOperation.java:331)
at com.google.refine.operations.column.ColumnAdditionByFetchingURLsOperation$ColumnAdditionByFetchingURLsProcess.run(ColumnAdditionByFetchingURLsOperation.java:292)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
Expected behavior
Either not run out of memory ;) or at least show an error message in the interface.
Desktop (please complete the following information):
- OS: Ubuntu 18.04.3 LTS
- Browser Version: Chrome Version 78.0.3904.87 (Official Build) (64-bit)
- JRE or JDK Version:
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build 1.8.0_232-8u232-b09-0ubuntu1~18.04.1-b09)
OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)
OpenRefine (please complete the following information):
- Version 3.3 Beta
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with ColumnAdditionByFetchingURLsOperation.java, especially cachedFetch and run, using the stack trace and the 30k-row reproduction as entry points. Investigate how the operation reports failures to the interface; done means the large fetch either avoids the OutOfMemoryError or presents an error instead of remaining stuck at 98–99%.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100