Codeception / Codeception/Specify
Failed test can't be re runned if the failed test use Codeception/Specify
- Ngôn ngữ chính
- PHP
- Star
- 158
- Fork
- 24
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
#### What are you trying to achieve?
Re run the failed tests:
```
./codecept run -g failed
```
#### What do you get instead?
Failed tests that use `Codeception/Specify` won't run. This problem only affects these tests.
#### How to reproduce
Create test file:
``` php
class FailedTest extends \Codeception\Test\Unit
{
use \Codeception\Specify;
public function testWithSpecify()
{
$this->specify('Specification', function () {
$this->assertFalse(true, 'Message 1');
});
}
public function testWithoutSpecify()
{
$this->assertFalse(true, 'Message 2');
}
}
```
Run tests:
```
$ ./codecept run
Codeception PHP Testing Framework v2.2.3
Powered by PHPUnit 5.4.8 by Sebastian Bergmann and contributors.
Acceptance Tests (0) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Functional Tests (0) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Unit Tests (2) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
✖ FailedTest: With specify | specification | examples index 0
✔ FailedTest: With specify (0.01s)
✖ FailedTest: Without specify (0.00s)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Time: 167 ms, Memory: 8.00MB
There were 2 failures:
---------
1) FailedTest: With specify | specification | examples index 0
Test tests/unit/FailedTest.php:testWithSpecify | Specification | examples index 0
Message 1
Failed asserting that true is false.
Codeception/Codeception#1 /mnt/store-ssd/work/codeception/tests/unit/FailedTest.php:11
Codeception/Codeception#2 FailedTest->{closure}
Codeception/Codeception#3 /mnt/store-ssd/work/codeception/tests/unit/FailedTest.php:12
Codeception/Codeception#4 FailedTest->testWithSpecify
---------
2) FailedTest: Without specify
Test tests/unit/FailedTest.php:testWithoutSpecify
Message 2
Failed asserting that true is false.
Codeception/Codeception#1 /mnt/store-ssd/work/codeception/tests/unit/FailedTest.php:17
Codeception/Codeception#2 FailedTest->testWithoutSpecify
FAILURES!
Tests: 2, Assertions: 2, Failures: 2.
```
`_output/failed` now contains the tests:
```
tests/unit/FailedTest.php:testWithSpecify | Specification | examples index 0
tests/unit/FailedTest.php:testWithoutSpecify
```
Run failed tests:
```
$ ./codecept run -g failed
Codeception PHP Testing Framework v2.2.3
Powered by PHPUnit 5.4.8 by Sebastian Bergmann and contributors.
[Groups] failed
Acceptance Tests (0) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Functional Tests (0) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Unit Tests (1) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
✖ FailedTest: Without specify (0.01s)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Time: 195 ms, Memory: 8.00MB
There was 1 failure:
---------
1) FailedTest: Without specify
Test tests/unit/FailedTest.php:testWithoutSpecify
Message 2
Failed asserting that true is false.
Codeception/Codeception#1 /mnt/store-ssd/work/codeception/tests/unit/FailedTest.php:17
Codeception/Codeception#2 FailedTest->testWithoutSpecify
FAILURES!
Tests: 1, Assertions: 1, Failures: 1.
```
**Note**: Only `testWithoutSpecify` runned.
#### Possible solution
I did some digging, and maybe I found the problem. I think that in [Lib\GroupManager](https://github.com/Codeception/Codeception/blob/2.2/src/Codeception/Lib/GroupManager.php#L121) an `strpos()` call's parameters are swapped.
The `$testPattern` contains: `/mnt/store-ssd/work/codeception/tests/unit/FailedTest.php:testWithSpecify | Specification | examples index 0`.
And `$filename . ':' . $test->getName(false)` is: `/mnt/store-ssd/work/codeception/tests/unit/FailedTest.php:testWithSpecify`.
So the `strpos()` returns `false`. But if I swap the two parameters, the all the failed tests are re runned.
I am new to Codeception, so I might be wrong (I haven't set up the tests, so I can't verify that this change doesn't break anything).
### Details
- Codeception version: 2.2.3
- PHP Version: 7.0.8
- Operating System: Ubuntu 16.04.1 LTS
- Installation type: Composer
- List of installed packages (`composer show`)
```
behat/gherkin v4.4.1 Gherkin DSL parser for PHP 5.3
codeception/codeception 2.2.3 BDD-style testing framework
codeception/specify 0.4.3 BDD code blocks for PHPUnit and Codeception
doctrine/instantiator 1.0.5 A small, lightweight utility to instantiate objects in PHP without invoking their constructors
facebook/webdriver 1.1.2 A PHP client for WebDriver
guzzlehttp/guzzle 6.2.1 Guzzle is a PHP HTTP client library
guzzlehttp/promises 1.2.0 Guzzle promises library
guzzlehttp/psr7 1.3.1 PSR-7 message implementation
myclabs/deep-copy 1.5.1 Create deep copies (clones) of your objects
phpdocumentor/reflection-common 1.0 Common reflection classes used by phpdocumentor to reflect the code structure
phpdocumentor/reflection-docblock 3.1.0 With this component, a library can provide support for annotations via DocBlocks or otherwise retrieve information that is embedded in a DocBlock.
phpdocumentor/type-resolver 0.2
phpspec/prophecy v1.6.1 Highly opinionated mocking framework for PHP 5.3+
phpunit/php-code-coverage 4.0.1 Library that provides collection, processing, and rendering functionality for PHP code coverage information.
phpunit/php-file-iterator 1.4.1 FilterIterator implementation that filters files based on a list of suffixes.
phpunit/php-text-template 1.2.1 Simple template engine.
phpunit/php-timer 1.0.8 Utility class for timing
phpunit/php-token-stream 1.4.8 Wrapper around PHP's tokenizer extension.
phpunit/phpunit 5.4.8 The PHP Unit Testing framework.
phpunit/phpunit-mock-objects 3.2.3 Mock Object library for PHPUnit
psr/http-message 1.0 Common interface for HTTP messages
sebastian/code-unit-reverse-lookup 1.0.0 Looks up which function or method a line of code belongs to
sebastian/comparator 1.2.0 Provides the functionality to compare PHP values for equality
sebastian/diff 1.4.1 Diff implementation
sebastian/environment 1.3.7 Provides functionality to handle HHVM/PHP environments
sebastian/exporter 1.2.2 Provides the functionality to export PHP variables for visualization
sebastian/global-state 1.1.1 Snapshotting of global state
sebastian/object-enumerator 1.0.0 Traverses array structures and object graphs to enumerate all referenced objects
sebastian/recursion-context 1.0.2 Provides functionality to recursively process PHP variables
sebastian/resource-operations 1.0.0 Provides a list of PHP built-in functions that operate on resources
sebastian/version 2.0.0 Library that helps with managing the version number of Git-hosted PHP projects
symfony/browser-kit v3.1.3 Symfony BrowserKit Component
symfony/console v3.1.3 Symfony Console Component
symfony/css-selector v3.1.3 Symfony CssSelector Component
symfony/dom-crawler v3.1.3 Symfony DomCrawler Component
symfony/event-dispatcher v3.1.3 Symfony EventDispatcher Component
symfony/finder v3.1.3 Symfony Finder Component
symfony/polyfill-mbstring v1.2.0 Symfony polyfill for the Mbstring extension
symfony/yaml v3.1.3 Symfony Yaml Component
webmozart/assert 1.0.2 Assertions to validate method input/output with nice error messages.
```
- Suite configuration:
I have set up a new project to demonstrate this. I've installed Codeception and Specify with composer (`composer require codeception/codeception codeception/specify`), and I've run bootstrap: `./codecept bootstrap`.
So, there is nothing special, default installation:
``` yml
actor: Tester
paths:
tests: tests
log: tests/_output
data: tests/_data
support: tests/_support
envs: tests/_envs
settings:
bootstrap: _bootstrap.php
colors: true
memory_limit: 1024M
extensions:
enabled:
- Codeception\Extension\RunFailed
modules:
config:
Db:
dsn: ''
user: ''
password: ''
dump: tests/_data/dump.sql
```
``` yml
# Codeception Test Suite Configuration
#
# Suite for unit (internal) tests.
class_name: UnitTester
modules:
enabled:
- Asserts
- \Helper\Unit
```
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Hướng nghiên cứu
Bắt đầu trong Lib\GroupManager.php tại lệnh gọi strpos() được issue đề cập, sau đó tái hiện lỗi bằng ./codecept run -g failed với một test có chứa Codeception\Specify. Hoàn tất khi nhóm failed-test chạy lại cả các lỗi Specify và non-Specify được liệt kê trong _output/failed.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- php
- Lĩnh vực
- testing
- Loại issue
- Lỗi
- Độ khó
- 2/5
- Thời gian dự kiến
- 1-3 giờ
- Mức độ hoạt động
- Đình trệ
- Độ rõ ràng
- Đặc tả rõ ràng
- Mức phù hợp với người mới
- 35/100