magento / magento/magento2-functional-testing-framework

Unable to generate Test files

Open
#884 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
PHP
Stars
154
Forks
134
PR merge metrics
No merged PRs in 30d

Description

### Preconditions

* php 7.4
* mftf version 3.5.1
* magento version 2.4.3
* Selenium driver 3.141.59 with chromium 98.0.4758.9

### Steps to reproduce

I have a test which looks like this:

```xml






<description value="Should be able to place an order as a Guest"/>
<severity value="CRITICAL"/>
<testCaseId value="internal-66062"/>
<group value="checkout"/>
</annotations>
<amOnPage url="{{HandballFuturaProduct.url}}" stepKey="onProductPage"/>
<waitForPageLoad stepKey="waitForPageLoad1"/>
<click selector="button[form=product_addtocart_form]" stepKey="addToCart"/>
<waitForElementVisible selector="{{StorefrontCategoryMainSection.SuccessMsg}}" time="30" stepKey="waitForProductAdded"/>
<see selector="{{StorefrontCategoryMainSection.SuccessMsg}}" userInput="Sie haben {{HandballFuturaProduct.name}} zu ihrem Warenkorb hinzugefügt." stepKey="seeAddedToCartMessage"/>
<see selector="{{StorefrontProductDetailSection.quantity}}" userInput="1" stepKey="seeCartQuantity"/>
</test>
</tests>
```
The test is placed in `dev/tests/acceptance/tests/functional/Magento/FunctionalTest/CompanyGeneral/Test/StorefrontGuestCheckoutHandballTest.xml`.

And a test suite which contains it:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<suites xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="../../../../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Suite/etc/suiteSchema.xsd">
<suite name="Company_Checkout">
<before>
<magentoCLI stepKey="disableAdminSecurityKeyInUrl" command="config:set admin/security/use_form_key 0" />
<magentoCLI stepKey="enableAdminAccountSharing" command="config:set admin/security/admin_account_sharing 1" />
<magentoCLI stepKey="disableWYSIWYGEditor" command="config:set cms/wysiwyg/enabled disabled" />
<magentoCLI stepKey="disableRecaptcha" command="config:set msp_securitysuite_recaptcha/frontend/enabled 0"/>
<magentoCLI stepKey="setIndexModeToRealtime" command="indexer:set-mode realtime" />
<magentoCLI stepKey="flushCache1" command="cache:flush" />
</before>
<include>
<test name="StorefrontGuestCheckoutHandballTest"/>
</include>
<after>
<magentoCLI stepKey="enableAdminSecurityKeyInUrl" command="config:set admin/security/use_form_key 1" />
<magentoCLI stepKey="disableAdminAccountSharing" command="config:set admin/security/admin_account_sharing 0" />
<magentoCLI stepKey="hideWYSIWYGEditor" command="config:set cms/wysiwyg/enabled hidden" />
<magentoCLI stepKey="enableRecaptcha" command="config:set msp_securitysuite_recaptcha/frontend/enabled 1"/>
<magentoCLI stepKey="setIndexModeToSchedule" command="indexer:set-mode schedule design_config_grid customer_grid catalog_category_product catalog_product_category catalogrule_rule catalog_product_attribute cataloginventory_stock catalogsearch_fulltext" />
<magentoCLI stepKey="flushCache2" command="cache:flush" />
</after>
</suite>
</suites>
```

The testsuite is saved in `dev/tests/acceptance/tests/_suite/CompanyCheckoutSuite.xml`.

I have `MODULE_ALLOWLIST=Magento_Framework,CompanyGeneral` set in my `.env`.

Trying to generate the suite does not work:

```
$ vendor/bin/mftf generate:suite Company_Checkout --remove

NOTICE: 287 Section name violations detected. See mftf.log for details.[]
NOTICE: 741 section element name violations detected. See mftf.log for details.[]
NOTICE: 372 data entity name violations detected. See mftf.log for details.[]
NOTICE: 11 data entity key violations detected. See mftf.log for details.[]
NOTICE: 441 action group name violations detected. See mftf.log for details.[]
NOTICE: 45 Page name violations detected. See mftf.log for details.[]
ERROR: 1 Suite(s) has(have) tests with annotation errors or some included tests missing. See mftf.log for details.
ERROR: 17 Test(s) failed to generate. See mftf.log for details.

Suites Generated (with errors)
```

There's so many log entries I don't know what to do about this. Pretty much all of them seem unrelated to my test suite or the test I created.
[Here is the full log](https://gist.github.com/kolaente/55778d78da16eff86b65d9bdbf51a0d7)

Trying to run the suite runs a million other tests, but not the one test specified in the suite:

```
$ vendor/bin/mftf run:group Company_Checkout --remove

ERROR: 1 Suite(s) has(have) tests with annotation errors or some included tests missing. See mftf.log for details.
ERROR: 17 Test(s) failed to generate. See mftf.log for details.

Generate Tests Command Run (with errors)

Codeception PHP Testing Framework v4.1.28
Powered by PHPUnit 9.1.5 by Sebastian Bergmann and contributors.
[Groups] Kuebler_Checkout

Magento\FunctionalTestingFramework.functional Tests (3567) ---------------------
Modules: \Magento\FunctionalTestingFramework\Module\MagentoWebDriver, \Magento\FunctionalTestingFramework\Module\MagentoSequence, \Magento\FunctionalTestingFramework\Module\MagentoAssert, \Magento\FunctionalTestingFramework\Module\MagentoActionProxies, Asserts, \Magento\FunctionalTestingFramework\Helper\HelperContainer
--------------------------------------------------------------------------------

.... more test runs omitted....

```

The logs only tell me it worked:

```
[2022-01-28 15:43:23] Magento\FunctionalTestingFramework\Suite\SuiteGenerator.INFO: suite generated {"suite":"Company_Checkout","relative_path":"_generated/Company_Checkout"} []
```

[Full logs here](https://gist.github.com/kolaente/d3bcf531e1d0732047d74c7a341b6862).

Trying to run only the test (without the suite) doesn't work either:

```
$ vendor/bin/mftf run:test StorefrontGuestCheckoutHandballTest --remove

ERROR: 1 Suite(s) has(have) tests with annotation errors or some included tests missing. See mftf.log for details.
ERROR: 1 Test(s) failed to generate. See mftf.log for details.

Generate Tests Command Run (with errors)

In RunTestCommand.php line 152:

StorefrontGuestCheckoutHandballTestCest.php is not available under /home/konrad/projects/company.de/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/default/

run:test [-k|--skip-generate] [-r|--remove] [-f|--force] [-a|--allow-skipped] [-d|--debug [DEBUG]] [--] <name>...
```

This time though, the logs say

```
[2022-01-28 15:46:58] Magento\FunctionalTestingFramework\Test\Handlers\TestObjectHandler.ERROR: Line 289: Test StorefrontGuestCheckoutHandballTest not defined in xml. [] []
[2022-01-28 15:46:58] Magento\FunctionalTestingFramework\Suite\Util\SuiteObjectExtractor.ERROR: Unable to find <test> reference StorefrontGuestCheckoutHandballTest for suite StorefrontGuestCheckoutHandballTest [] []
```

but the test _does_ exist, the mftf just doesn't find it??

[Full log](https://gist.github.com/kolaente/3a7cbe2e71176d0618d7b75e9d7489c4).

Honestly, at this point I'm pretty frustrated. I've tried for a few hours to get this to work, tried a few different configurations and nothing worked.

### Expected result

Tests run or at least tell me what's wrong.

### Actual result

Tests don't run.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the test XML at dev/tests/acceptance/tests/functional/Magento/FunctionalTest/CompanyGeneral/Test/StorefrontGuestCheckoutHandballTest.xml and the suite at dev/tests/acceptance/tests/_suite/CompanyCheckoutSuite.xml. Run the reported generate:suite and run:test commands, then trace mftf.log entries for annotation errors, missing tests, and the generated path. Done means the named test or suite generates and runs without unrelated tests or unexplained errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
cli, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.