rust-lang / rust-lang/libs-team

ACP: give access to the inner iterator of `std::iter::Peekable`

Open
#802 2 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

api-change-proposal
Dominant language
Rust
Stars
178
Forks
28
Avg merge
15m
Merged PRs (30d)
1

Description

Proposal

Problem statement

Some iterators carry some additional state that's not encompassed by the Iterator trait, e.g. an additional method or a field, but after wrapping them in Peekable, there's no more access to anything in them besides their Iterator impl.

Motivating examples or use cases

There's a Lexer struct, that implements Iterator by handing out Lexeme objects. One way to return lexing errors could be to yield Result<Lexeme, LexingError> via its Iterator impl, but that makes it very inconvenient to create processing pipelines out of this iterator, having to check for an error on every iteration. Instead, the Lexer struct yield just Lexeme, and stores the error in itself, which is to be retrieved after exhausting the iterator with Lexer::finish

struct LexingError { /* ... */ }
struct Lexeme { /* ... */ }
struct Lexer {
    res: Result<(), LexingError>,
    // ...
}

impl Iterator for Lexer {
    type Item = Lexeme;

    fn next(&mut self) -> Option<Self::Item> {
        todo!()
    }
}

impl Lexer {
    fn finish(self) -> Result<(), LexingError> {
        self.res
    }
}

It could be useful to wrap the Lexer in Peekable to create a parser that tries various paths depending on the next token, but then one wouldn't be able to retrieve the error, and have to opt for the clumsier approach with type Item = Result<Lexeme, LexingError>;

Solution sketch

impl<I> Peekable<I> {
    fn inner(&self) -> &I;
    fn into_inner(self) -> I;
}

No inner_mut as it'd allow for scrambling the order of the items yielded by the peekable iterator

Alternatives

  1. Don't put any additional state into iterators that are to be made Peekable
    That's not always possible since one doesn't always have control over how the state is stored by various objects.

  2. Create your own Peekable that allows for accessing the inner iterator
    That's what I did in my own project, but this approach leads to a ton of boilerplate & likely won't allow for an implementation as performant & flexible as the one provided by the standard library

Links and related work

std::path::Components, std::path::Iter & std::str::Chars all have methods that return the remainder of the data to be processed.

What happens now?

This issue contains an API change proposal (or ACP) and is part of the libs-api team feature lifecycle. Once this issue is filed, the libs-api team will review open proposals as capability becomes available. Current response times do not have a clear estimate, but may be up to several months.

Possible responses

The libs team may respond in various different ways. First, the team will consider the problem (this doesn't require any concrete solution or alternatives to have been proposed):

  • We think this problem seems worth solving, and the standard library might be the right place to solve it.
  • We think that this probably doesn't belong in the standard library.

Second, if there's a concrete solution:

  • We think this specific solution looks roughly right, approved, you or someone else should implement this. (Further review will still happen on the subsequent implementation PR.)
  • We're not sure this is the right solution, and the alternatives or other materials don't give us enough information to be sure about that. Here are some questions we have that aren't answered, or rough ideas about alternatives we'd want to see discussed.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the std::iter::Peekable API context, the Lexer::finish motivating example, and the linked feature lifecycle. No repository files or tests are named; completion depends on libs-api review of the proposed API and a later implementation decision.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.