JuliaData / JuliaData/DataFramesMeta.jl

Very slow `@astable` macro outside a function

Open
#363 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
496
Forks
56
PR merge metrics
No merged PRs in 30d

Description

Here is the experiment.

Given the dataframe and functions f0, f1 below

using DataFrames, DataFramesMeta, StatsBase
df = DataFrame(a=1:10_000)  # I know the df is small but big enough to show the issue
f0(df::DataFrame) = begin
	@chain df begin
		@rtransform(:b = :a * 10)
		@rtransform(:c = mean(:b))
		@rtransform(:d = :b - :c)
		@select(:a, :d)
	end
end
f1(df::DataFrame) = begin
	@chain df begin
		@rtransform @astable begin
			b = :a * 10
			c = mean(b)
			:d = b - c
		end
	end
end

We get an improvement in performance in f1, which is what one would expect given it does not need to create columns b, c .

@time f0(df)
0.001146 seconds (728 allocations: 898.516 KiB)
@time f1(df)
0.000503 seconds (161 allocations: 243.609 KiB)

However, if one uses this code outside a function (see below) it becomes 46 times slower! Making it unusable for datasets of a larger size.

@time @chain df begin
	@rtransform @astable begin
		b = :a * 10
		c = mean(b)
		:d = b - c
	end
end
->  2.331518 seconds (335.93 k allocations: 13.028 MiB, 4.69% compilation time)

@time @chain df begin
	@rtransform(:b = :a * 10)
	@rtransform(:c = mean(:b))
	@rtransform(:d = :b - :c)
	@select(:a, :d)
end
->  0.056910 seconds (34.81 k allocations: 3.137 MiB, 95.06% compilation time)

Thanks for the great work :)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the two @chain experiments in the issue, comparing the @astable form outside a function with the equivalent chained transformations. Trace the @astable and @rtransform macro expansions to identify why top-level execution allocates and compiles so much more. Done means the top-level form no longer has the reported severe slowdown while preserving the function behavior and output.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.