byexamples / byexamples/byexample

Use last line of the expected output to sync with the interpreter in addition to the prompt

Open
#181 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
67
Forks
9
PR merge metrics
No merged PRs in 30d

Description

**Describe the feature you'd like**
`byexample` *expects* the main prompt (PS1) to know when an example finished, recollects all the output *before* the prompt and compares it with the expected output for that example.

This reasonably assumes that all the output *arrives before* the prompt and *nothing arrives after*.

But not always is the case: an example may run in background and the interpreter may print the prompt before the example finishes. The result would be the last part of the output of the example may be missing and `byexample` will mark the example as failed but it also the missing output may appear at the begin of the *next example* output, unexpected for it, and `byexample` will mark it as failed too.

In #176 we added a `-x-delayafterprompt` to wait a fixed amount of time and try to read any output *after* the prompt but a fixed *sleep* isn't good for the performance and it is fragile anyways.

`byexample` could use the example's expected last lines as part of the synchronization.

### Proposed solution

We could expect the prompt as usual and then check if the last expected line is in the output read. If isn't, try to do a second expect for that line and add any new output to the already read.

This extra check may fail due a timeout if:
1. the output never arrives and we are still out of sync
2. there is no output at all because the example is really printing something totally different

We cannot do much for 1 except increase its timeout. But that will go against 2: the second case is the *happy case* where we are not out of sync (but the example still fails) and increasing the timeout of the extra check will slowdown the execution.

We also needs to define which it means "last line". Which would be the minimum length for such thing to be useful for synchronization?.

**Additional context (optional)**
This issue probably a hard-dependency for #176 because `evcxr`, the `rust` interpreter redirects the examples' outputs (`stdout` and `stderr`) to the terminal in background. This seems to affect only when the example does `println!()`-like something so it is not going to be the most common case but certainly if the user hit the issue it will be confusing for him/her.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.