queryverse / queryverse/Query.jl

long wait times for first run of `@map` and secondary problem of REPL hanging.

Open
#232 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug upstream
Dominant language
Julia
Stars
403
Forks
48
Avg merge
3d 6h
Merged PRs (30d)
6

Description

I can only reproduce this with my data file which is hosted on github link. It's not a large file (270kb, 250 rows by 500 columns) of integer data, delimited by tab. Please download this file and save as symp.dat (or alter the code below accordingly).

using DataFrames
using Statistics
using Query
using Base.Filesystem
using CSV

headers = ["sim$i" for i = 1:500]  
dt = CSV.File("symp.dat", delim='\t', header=headers) |> DataFrame
dt.time = 1:250 ## add a time column for `melt` purposes

f(g) = g |> @map(mean(_))
@time f(dt)

Output:

julia> @time f(dt)
 31.975361 seconds (7.63 M allocations: 415.528 MiB, 0.84% gc time)

And then the REPL hangs. It hangs for about 2 minutes! before it finishes printing the 250-element query result.

julia> @time f(dt)
 31.190349 seconds (7.64 M allocations: 416.400 MiB, 0.94% gc time)

250-element query result
 0.249501
 0.265469
 0.0838323
 0.237525
 0.323353
 0.359281
 0.457086
 0.566866
 0.798403
 1.06587
... with 240 more elements

In summary:

  • First run takes about 35 seconds even when @map is in a function (maybe this has to something with the fact that dtis global?)
  • The REPL hangs for about 2 minutes after the @time macro prints its information. It seems to me it takes a long time to "collect" the results of the query and display it back in the REPL.

Version info:

julia> versioninfo()
Julia Version 1.0.0
Commit 5d4eaca0c9 (2018-08-08 20:58 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin14.5.0)
  CPU: Intel(R) Core(TM) i7-3615QM CPU @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, ivybridge)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Download symp.dat and reproduce the example using DataFrames, CSV, and Query with f(dt) and @map(mean(_)). Measure the initial @map run separately from the REPL's printing of the 250-element query result. Done means identifying and addressing the reported first-run delay and post-timing REPL hang, with the example no longer exhibiting those waits.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.