magento / magento/magento2-functional-testing-framework
Unable to generate Test files
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 154
- Forks
- 134
- PR merge metrics
- No merged PRs in 30d
Description
### Preconditions
* php 7.4
* mftf version 3.5.1
* magento version 2.4.3
* Selenium driver 3.141.59 with chromium 98.0.4758.9
### Steps to reproduce
I have a test which looks like this:
```xml
<description value="Should be able to place an order as a Guest"/>
<severity value="CRITICAL"/>
<testCaseId value="internal-66062"/>
<group value="checkout"/>
</annotations>
<amOnPage url="{{HandballFuturaProduct.url}}" stepKey="onProductPage"/>
<waitForPageLoad stepKey="waitForPageLoad1"/>
<click selector="button[form=product_addtocart_form]" stepKey="addToCart"/>
<waitForElementVisible selector="{{StorefrontCategoryMainSection.SuccessMsg}}" time="30" stepKey="waitForProductAdded"/>
<see selector="{{StorefrontCategoryMainSection.SuccessMsg}}" userInput="Sie haben {{HandballFuturaProduct.name}} zu ihrem Warenkorb hinzugefügt." stepKey="seeAddedToCartMessage"/>
<see selector="{{StorefrontProductDetailSection.quantity}}" userInput="1" stepKey="seeCartQuantity"/>
</test>
</tests>
```
The test is placed in `dev/tests/acceptance/tests/functional/Magento/FunctionalTest/CompanyGeneral/Test/StorefrontGuestCheckoutHandballTest.xml`.
And a test suite which contains it:
```xml
<?xml version="1.0" encoding="UTF-8"?>
<suites xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="../../../../../vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Suite/etc/suiteSchema.xsd">
<suite name="Company_Checkout">
<before>
<magentoCLI stepKey="disableAdminSecurityKeyInUrl" command="config:set admin/security/use_form_key 0" />
<magentoCLI stepKey="enableAdminAccountSharing" command="config:set admin/security/admin_account_sharing 1" />
<magentoCLI stepKey="disableWYSIWYGEditor" command="config:set cms/wysiwyg/enabled disabled" />
<magentoCLI stepKey="disableRecaptcha" command="config:set msp_securitysuite_recaptcha/frontend/enabled 0"/>
<magentoCLI stepKey="setIndexModeToRealtime" command="indexer:set-mode realtime" />
<magentoCLI stepKey="flushCache1" command="cache:flush" />
</before>
<include>
<test name="StorefrontGuestCheckoutHandballTest"/>
</include>
<after>
<magentoCLI stepKey="enableAdminSecurityKeyInUrl" command="config:set admin/security/use_form_key 1" />
<magentoCLI stepKey="disableAdminAccountSharing" command="config:set admin/security/admin_account_sharing 0" />
<magentoCLI stepKey="hideWYSIWYGEditor" command="config:set cms/wysiwyg/enabled hidden" />
<magentoCLI stepKey="enableRecaptcha" command="config:set msp_securitysuite_recaptcha/frontend/enabled 1"/>
<magentoCLI stepKey="setIndexModeToSchedule" command="indexer:set-mode schedule design_config_grid customer_grid catalog_category_product catalog_product_category catalogrule_rule catalog_product_attribute cataloginventory_stock catalogsearch_fulltext" />
<magentoCLI stepKey="flushCache2" command="cache:flush" />
</after>
</suite>
</suites>
```
The testsuite is saved in `dev/tests/acceptance/tests/_suite/CompanyCheckoutSuite.xml`.
I have `MODULE_ALLOWLIST=Magento_Framework,CompanyGeneral` set in my `.env`.
Trying to generate the suite does not work:
```
$ vendor/bin/mftf generate:suite Company_Checkout --remove
NOTICE: 287 Section name violations detected. See mftf.log for details.[]
NOTICE: 741 section element name violations detected. See mftf.log for details.[]
NOTICE: 372 data entity name violations detected. See mftf.log for details.[]
NOTICE: 11 data entity key violations detected. See mftf.log for details.[]
NOTICE: 441 action group name violations detected. See mftf.log for details.[]
NOTICE: 45 Page name violations detected. See mftf.log for details.[]
ERROR: 1 Suite(s) has(have) tests with annotation errors or some included tests missing. See mftf.log for details.
ERROR: 17 Test(s) failed to generate. See mftf.log for details.
Suites Generated (with errors)
```
There's so many log entries I don't know what to do about this. Pretty much all of them seem unrelated to my test suite or the test I created.
[Here is the full log](https://gist.github.com/kolaente/55778d78da16eff86b65d9bdbf51a0d7)
Trying to run the suite runs a million other tests, but not the one test specified in the suite:
```
$ vendor/bin/mftf run:group Company_Checkout --remove
ERROR: 1 Suite(s) has(have) tests with annotation errors or some included tests missing. See mftf.log for details.
ERROR: 17 Test(s) failed to generate. See mftf.log for details.
Generate Tests Command Run (with errors)
Codeception PHP Testing Framework v4.1.28
Powered by PHPUnit 9.1.5 by Sebastian Bergmann and contributors.
[Groups] Kuebler_Checkout
Magento\FunctionalTestingFramework.functional Tests (3567) ---------------------
Modules: \Magento\FunctionalTestingFramework\Module\MagentoWebDriver, \Magento\FunctionalTestingFramework\Module\MagentoSequence, \Magento\FunctionalTestingFramework\Module\MagentoAssert, \Magento\FunctionalTestingFramework\Module\MagentoActionProxies, Asserts, \Magento\FunctionalTestingFramework\Helper\HelperContainer
--------------------------------------------------------------------------------
.... more test runs omitted....
```
The logs only tell me it worked:
```
[2022-01-28 15:43:23] Magento\FunctionalTestingFramework\Suite\SuiteGenerator.INFO: suite generated {"suite":"Company_Checkout","relative_path":"_generated/Company_Checkout"} []
```
[Full logs here](https://gist.github.com/kolaente/d3bcf531e1d0732047d74c7a341b6862).
Trying to run only the test (without the suite) doesn't work either:
```
$ vendor/bin/mftf run:test StorefrontGuestCheckoutHandballTest --remove
ERROR: 1 Suite(s) has(have) tests with annotation errors or some included tests missing. See mftf.log for details.
ERROR: 1 Test(s) failed to generate. See mftf.log for details.
Generate Tests Command Run (with errors)
In RunTestCommand.php line 152:
StorefrontGuestCheckoutHandballTestCest.php is not available under /home/konrad/projects/company.de/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/default/
run:test [-k|--skip-generate] [-r|--remove] [-f|--force] [-a|--allow-skipped] [-d|--debug [DEBUG]] [--] <name>...
```
This time though, the logs say
```
[2022-01-28 15:46:58] Magento\FunctionalTestingFramework\Test\Handlers\TestObjectHandler.ERROR: Line 289: Test StorefrontGuestCheckoutHandballTest not defined in xml. [] []
[2022-01-28 15:46:58] Magento\FunctionalTestingFramework\Suite\Util\SuiteObjectExtractor.ERROR: Unable to find <test> reference StorefrontGuestCheckoutHandballTest for suite StorefrontGuestCheckoutHandballTest [] []
```
but the test _does_ exist, the mftf just doesn't find it??
[Full log](https://gist.github.com/kolaente/3a7cbe2e71176d0618d7b75e9d7489c4).
Honestly, at this point I'm pretty frustrated. I've tried for a few hours to get this to work, tried a few different configurations and nothing worked.
### Expected result
Tests run or at least tell me what's wrong.
### Actual result
Tests don't run.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the test XML at dev/tests/acceptance/tests/functional/Magento/FunctionalTest/CompanyGeneral/Test/StorefrontGuestCheckoutHandballTest.xml and the suite at dev/tests/acceptance/tests/_suite/CompanyCheckoutSuite.xml. Run the reported generate:suite and run:test commands, then trace mftf.log entries for annotation errors, missing tests, and the generated path. Done means the named test or suite generates and runs without unrelated tests or unexplained errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- php
- Domain
- cli, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100