matomo-org / matomo-org/plugin-Migration
Very high memory usage when migrating measurable
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 7
- Forks
- 11
- Avg merge
- 4h 17m
- Merged PRs (30d)
- 2
Description
We had another user that had a similar experience as report in #38
However, in this case there was at least an error that was output:
`ERROR [2022-08-01 11:12:19] 348340 Fatal error encountered: /var/www/matomo/libs/Zend/Db/Statement/Pdo.php(292): Allowed memory of 6442450944 bytes exhausted (tried to allocate 8192 bytes) [Query: , CLI mode: 1]`
The user needed to increase the allowed memory to all available memory (AFAIK 16gb at least) in order for the migration to complete.
I asked the user to send us the size of the log_ tables, since it seems that there is an array created and held in memory for `LogActionMigration` which might be kept for the entire Migration.
But the size of the tables for this database are not very large, so the used memory seems to be quite excessive:
```
+--------------------+----------------------------------------------------+------------+
| Database | Table | Size in MB |
+--------------------+----------------------------------------------------+------------+
| MatomoDBName | log_link_visit_action | 1418.98 |
| MatomoDBName | log_visit | 1310.73 |
| MatomoDBName | log_action | 12.03 |
| MatomoDBName | log_media | 2.80 |
| MatomoDBName | log_media_plays | 0.08 |
| MatomoDBName | log_form | 0.05 |
| MatomoDBName | log_abtesting | 0.05 |
| MatomoDBName | log_hsr | 0.05 |
| MatomoDBName | log_hsr_event | 0.05 |
| MatomoDBName | log_conversion | 0.05 |
| MatomoDBName | log_hsr_blob | 0.03 |
| MatomoDBName | log_hsr_site | 0.03 |
| MatomoDBName | log_form_page | 0.03 |
| MatomoDBName | log_profiling | 0.03 |
| MatomoDBName | log_conversion_item | 0.03 |
| MatomoDBName | log_funnel | 0.03 |
| MatomoDBName | log_form_field | 0.02 |
+--------------------+----------------------------------------------------+------------+
```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the LogActionMigration path and the related report in issue #38, then reproduce the measurable migration with the reported table sizes and memory limit. Trace what migration data remains in memory during the process; done means the migration completes without requiring excessive memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- php
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100