byexamples / byexamples/byexample
Use last line of the expected output to sync with the interpreter in addition to the prompt
- Dominant language
- Python
- Stars
- 67
- Forks
- 9
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the feature you'd like**
`byexample` *expects* the main prompt (PS1) to know when an example finished, recollects all the output *before* the prompt and compares it with the expected output for that example.
This reasonably assumes that all the output *arrives before* the prompt and *nothing arrives after*.
But not always is the case: an example may run in background and the interpreter may print the prompt before the example finishes. The result would be the last part of the output of the example may be missing and `byexample` will mark the example as failed but it also the missing output may appear at the begin of the *next example* output, unexpected for it, and `byexample` will mark it as failed too.
In #176 we added a `-x-delayafterprompt` to wait a fixed amount of time and try to read any output *after* the prompt but a fixed *sleep* isn't good for the performance and it is fragile anyways.
`byexample` could use the example's expected last lines as part of the synchronization.
### Proposed solution
We could expect the prompt as usual and then check if the last expected line is in the output read. If isn't, try to do a second expect for that line and add any new output to the already read.
This extra check may fail due a timeout if:
1. the output never arrives and we are still out of sync
2. there is no output at all because the example is really printing something totally different
We cannot do much for 1 except increase its timeout. But that will go against 2: the second case is the *happy case* where we are not out of sync (but the example still fails) and increasing the timeout of the extra check will slowdown the execution.
We also needs to define which it means "last line". Which would be the minimum length for such thing to be useful for synchronization?.
**Additional context (optional)**
This issue probably a hard-dependency for #176 because `evcxr`, the `rust` interpreter redirects the examples' outputs (`stdout` and `stderr`) to the terminal in background. This seems to affect only when the example does `println!()`-like something so it is not going to be the most common case but certainly if the user hit the issue it will be confusing for him/her.
Contributor guide
Assessment
This issue has not been assessed yet.