apache / apache/datafusion

Add `tail` method on DataFrame

Open
#12,911 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

Is there a better way we could do this? Maybe add something upstream if necessary?

As I'm thinking of it, I don't know that this operation is necessarily well defined. Just like with `limit` when you call it multiple times on a large dataframe you get different results, I would expect different results from multiple calls here.

If we do put this in, I would suggest adding more text to the description to explain why this is an expensive operation - that it performs a collect to determine the size of the dataframe.

_Originally posted by @timsaucer in https://github.com/apache/datafusion-python/pull/915#discussion_r1798327215_

Contributor guide

Open the contributing guide

Research direction

Start with the discussion in datafusion-python PR #915, where this proposal originated. Clarify the semantics of repeated tail calls and the cost of collecting to determine DataFrame size before choosing an implementation. Done means an agreed tail API with documentation explaining its behavior and expense.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.