magento / magento/data-migration-tool

Cannot complete url re-write step of migration

Open
#700 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
PHP
Stars
339
Forks
195
PR merge metrics
No merged PRs in 30d

Description

### Preconditions

1. Data Migration from Magento Commerce 1.14.4.0 to Magento Commerce 2.3.1, using php-fpm 7.2.11 and mysql 5.7 (using nginx / varnish)
2. Magento 2 Database is fresh & recently re-created to ensure no unexpected data is present for the migration
3. Magento 2 database user has full privileges
4. M1 db is hosting in AWS and firewall rule is open so it can talk to our M2 docker container where we're testing the migration. Networking issues are not a problem.
5. M2 mysql server has 100M for max_allowed_packet and 15 minutes for default wait_timeout
6. Cron jobs are disabled on source and destination environments
7. We're able to complete the migration if we remove the URL Re-write step; just this step is causing issues.
8. ~150k SKUs being migrated over roughly ~40 categories and subcategories
9. I have tried this with batch_size being 1, 15, 50, 100, 250, and 400. No changes to this setting has helped. The url re-write step always dies after ~1 minute and 10 seconds.

### Steps to reproduce

1. You'd need a copy of our database (or a similarly sized catalog
2. Start the data migration tool
3. Get to the URL Re-write step

### Expected result

1. The URL-Rewrite step should begin (or complete)

### Actual result

1. data migration tool returns "Mysql Server has gone away" error after ~1 minute and 10 seconds, consistently.

### Additional notes
1. The mysql server has no useful errors or log messages. Nothing stating what could have caused the server to go away.
2. the migration tool / m2 server has no useful logging or errors
2. 100M should be more than enough for max_allowed_packet, based on what I've seen in google results / searching this github for related issues. But given that my company has such a large catalog / large amount of categories, could I possibly need to set it higher?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the migration with a similarly sized catalog and inspect the URL Re-write step where the MySQL server disconnects after about 1 minute and 10 seconds. Review the migration tool's logging and the MySQL settings mentioned, including batch_size, max_allowed_packet, and wait_timeout. Done means the URL Re-write step completes without the "Mysql Server has gone away" error.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, php
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.