matomo-org / matomo-org/plugin-Migration

Very high memory usage when migrating measurable

Open
#41 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
PHP
Stars
7
Forks
11
Avg merge
4h 17m
Merged PRs (30d)
2

Description

We had another user that had a similar experience as report in #38
However, in this case there was at least an error that was output:
`ERROR [2022-08-01 11:12:19] 348340 Fatal error encountered: /var/www/matomo/libs/Zend/Db/Statement/Pdo.php(292): Allowed memory of 6442450944 bytes exhausted (tried to allocate 8192 bytes) [Query: , CLI mode: 1]`

The user needed to increase the allowed memory to all available memory (AFAIK 16gb at least) in order for the migration to complete.
I asked the user to send us the size of the log_ tables, since it seems that there is an array created and held in memory for `LogActionMigration` which might be kept for the entire Migration.
But the size of the tables for this database are not very large, so the used memory seems to be quite excessive:
```
+--------------------+----------------------------------------------------+------------+
| Database | Table | Size in MB |
+--------------------+----------------------------------------------------+------------+
| MatomoDBName | log_link_visit_action | 1418.98 |
| MatomoDBName | log_visit | 1310.73 |
| MatomoDBName | log_action | 12.03 |
| MatomoDBName | log_media | 2.80 |
| MatomoDBName | log_media_plays | 0.08 |
| MatomoDBName | log_form | 0.05 |
| MatomoDBName | log_abtesting | 0.05 |
| MatomoDBName | log_hsr | 0.05 |
| MatomoDBName | log_hsr_event | 0.05 |
| MatomoDBName | log_conversion | 0.05 |
| MatomoDBName | log_hsr_blob | 0.03 |
| MatomoDBName | log_hsr_site | 0.03 |
| MatomoDBName | log_form_page | 0.03 |
| MatomoDBName | log_profiling | 0.03 |
| MatomoDBName | log_conversion_item | 0.03 |
| MatomoDBName | log_funnel | 0.03 |
| MatomoDBName | log_form_field | 0.02 |
+--------------------+----------------------------------------------------+------------+
```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the LogActionMigration path and the related report in issue #38, then reproduce the measurable migration with the reported table sizes and memory limit. Trace what migration data remains in memory during the process; done means the migration completes without requiring excessive memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.