Codeception / Codeception/Specify

Failed test can't be re runned if the failed test use Codeception/Specify

Abierto
#53 3 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
PHP
Estrellas
158
Forks
24
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

#### What are you trying to achieve?

Re run the failed tests:

```
./codecept run -g failed
```
#### What do you get instead?

Failed tests that use `Codeception/Specify` won't run. This problem only affects these tests.
#### How to reproduce

Create test file:

``` php
class FailedTest extends \Codeception\Test\Unit
{
use \Codeception\Specify;

public function testWithSpecify()
{
$this->specify('Specification', function () {
$this->assertFalse(true, 'Message 1');
});
}

public function testWithoutSpecify()
{
$this->assertFalse(true, 'Message 2');
}
}
```

Run tests:

```
$ ./codecept run
Codeception PHP Testing Framework v2.2.3
Powered by PHPUnit 5.4.8 by Sebastian Bergmann and contributors.

Acceptance Tests (0) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Functional Tests (0) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Unit Tests (2) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
✖ FailedTest: With specify | specification | examples index 0
✔ FailedTest: With specify (0.01s)
✖ FailedTest: Without specify (0.00s)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Time: 167 ms, Memory: 8.00MB

There were 2 failures:

---------
1) FailedTest: With specify | specification | examples index 0
Test tests/unit/FailedTest.php:testWithSpecify | Specification | examples index 0
Message 1
Failed asserting that true is false.
Codeception/Codeception#1 /mnt/store-ssd/work/codeception/tests/unit/FailedTest.php:11
Codeception/Codeception#2 FailedTest->{closure}
Codeception/Codeception#3 /mnt/store-ssd/work/codeception/tests/unit/FailedTest.php:12
Codeception/Codeception#4 FailedTest->testWithSpecify

---------
2) FailedTest: Without specify
Test tests/unit/FailedTest.php:testWithoutSpecify
Message 2
Failed asserting that true is false.
Codeception/Codeception#1 /mnt/store-ssd/work/codeception/tests/unit/FailedTest.php:17
Codeception/Codeception#2 FailedTest->testWithoutSpecify

FAILURES!
Tests: 2, Assertions: 2, Failures: 2.
```

`_output/failed` now contains the tests:

```
tests/unit/FailedTest.php:testWithSpecify | Specification | examples index 0
tests/unit/FailedTest.php:testWithoutSpecify
```

Run failed tests:

```
$ ./codecept run -g failed
Codeception PHP Testing Framework v2.2.3
Powered by PHPUnit 5.4.8 by Sebastian Bergmann and contributors.
[Groups] failed

Acceptance Tests (0) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Functional Tests (0) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Unit Tests (1) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
✖ FailedTest: Without specify (0.01s)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Time: 195 ms, Memory: 8.00MB

There was 1 failure:

---------
1) FailedTest: Without specify
Test tests/unit/FailedTest.php:testWithoutSpecify
Message 2
Failed asserting that true is false.
Codeception/Codeception#1 /mnt/store-ssd/work/codeception/tests/unit/FailedTest.php:17
Codeception/Codeception#2 FailedTest->testWithoutSpecify

FAILURES!
Tests: 1, Assertions: 1, Failures: 1.
```

**Note**: Only `testWithoutSpecify` runned.
#### Possible solution

I did some digging, and maybe I found the problem. I think that in [Lib\GroupManager](https://github.com/Codeception/Codeception/blob/2.2/src/Codeception/Lib/GroupManager.php#L121) an `strpos()` call's parameters are swapped.

The `$testPattern` contains: `/mnt/store-ssd/work/codeception/tests/unit/FailedTest.php:testWithSpecify | Specification | examples index 0`.
And `$filename . ':' . $test->getName(false)` is: `/mnt/store-ssd/work/codeception/tests/unit/FailedTest.php:testWithSpecify`.

So the `strpos()` returns `false`. But if I swap the two parameters, the all the failed tests are re runned.

I am new to Codeception, so I might be wrong (I haven't set up the tests, so I can't verify that this change doesn't break anything).
### Details
- Codeception version: 2.2.3
- PHP Version: 7.0.8
- Operating System: Ubuntu 16.04.1 LTS
- Installation type: Composer
- List of installed packages (`composer show`)

```
behat/gherkin v4.4.1 Gherkin DSL parser for PHP 5.3
codeception/codeception 2.2.3 BDD-style testing framework
codeception/specify 0.4.3 BDD code blocks for PHPUnit and Codeception
doctrine/instantiator 1.0.5 A small, lightweight utility to instantiate objects in PHP without invoking their constructors
facebook/webdriver 1.1.2 A PHP client for WebDriver
guzzlehttp/guzzle 6.2.1 Guzzle is a PHP HTTP client library
guzzlehttp/promises 1.2.0 Guzzle promises library
guzzlehttp/psr7 1.3.1 PSR-7 message implementation
myclabs/deep-copy 1.5.1 Create deep copies (clones) of your objects
phpdocumentor/reflection-common 1.0 Common reflection classes used by phpdocumentor to reflect the code structure
phpdocumentor/reflection-docblock 3.1.0 With this component, a library can provide support for annotations via DocBlocks or otherwise retrieve information that is embedded in a DocBlock.
phpdocumentor/type-resolver 0.2
phpspec/prophecy v1.6.1 Highly opinionated mocking framework for PHP 5.3+
phpunit/php-code-coverage 4.0.1 Library that provides collection, processing, and rendering functionality for PHP code coverage information.
phpunit/php-file-iterator 1.4.1 FilterIterator implementation that filters files based on a list of suffixes.
phpunit/php-text-template 1.2.1 Simple template engine.
phpunit/php-timer 1.0.8 Utility class for timing
phpunit/php-token-stream 1.4.8 Wrapper around PHP's tokenizer extension.
phpunit/phpunit 5.4.8 The PHP Unit Testing framework.
phpunit/phpunit-mock-objects 3.2.3 Mock Object library for PHPUnit
psr/http-message 1.0 Common interface for HTTP messages
sebastian/code-unit-reverse-lookup 1.0.0 Looks up which function or method a line of code belongs to
sebastian/comparator 1.2.0 Provides the functionality to compare PHP values for equality
sebastian/diff 1.4.1 Diff implementation
sebastian/environment 1.3.7 Provides functionality to handle HHVM/PHP environments
sebastian/exporter 1.2.2 Provides the functionality to export PHP variables for visualization
sebastian/global-state 1.1.1 Snapshotting of global state
sebastian/object-enumerator 1.0.0 Traverses array structures and object graphs to enumerate all referenced objects
sebastian/recursion-context 1.0.2 Provides functionality to recursively process PHP variables
sebastian/resource-operations 1.0.0 Provides a list of PHP built-in functions that operate on resources
sebastian/version 2.0.0 Library that helps with managing the version number of Git-hosted PHP projects
symfony/browser-kit v3.1.3 Symfony BrowserKit Component
symfony/console v3.1.3 Symfony Console Component
symfony/css-selector v3.1.3 Symfony CssSelector Component
symfony/dom-crawler v3.1.3 Symfony DomCrawler Component
symfony/event-dispatcher v3.1.3 Symfony EventDispatcher Component
symfony/finder v3.1.3 Symfony Finder Component
symfony/polyfill-mbstring v1.2.0 Symfony polyfill for the Mbstring extension
symfony/yaml v3.1.3 Symfony Yaml Component
webmozart/assert 1.0.2 Assertions to validate method input/output with nice error messages.
```
- Suite configuration:

I have set up a new project to demonstrate this. I've installed Codeception and Specify with composer (`composer require codeception/codeception codeception/specify`), and I've run bootstrap: `./codecept bootstrap`.

So, there is nothing special, default installation:

``` yml
actor: Tester
paths:
tests: tests
log: tests/_output
data: tests/_data
support: tests/_support
envs: tests/_envs
settings:
bootstrap: _bootstrap.php
colors: true
memory_limit: 1024M
extensions:
enabled:
- Codeception\Extension\RunFailed
modules:
config:
Db:
dsn: ''
user: ''
password: ''
dump: tests/_data/dump.sql
```

``` yml
# Codeception Test Suite Configuration
#
# Suite for unit (internal) tests.

class_name: UnitTester
modules:
enabled:
- Asserts
- \Helper\Unit
```

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.