Improve sqoop rucio table dumps with dynamic num-mappers
- Dominant language
- Go
- Stars
- 5
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
We are using sqoop `--num-mappers=40`[1] for all Rucio table dump queries. However, some tables like `rses, subscriptions` are so small tables and their run time can be decreased with `--num-mappers=1`.
Previous experiences showed that we can run 2 `--num-mappers-40` parallel, we did not implement it for Rucio table dumps since we don't have a time pressure. However, we can expect that our users will come with more Rucio table dump request, so we can implement it now.
So, my suggestions are:
- Run, at most, 2 Rucio table dumps parallel.
- Set --num-mappers=1 for "rses" and "subscriptions" tables.
P.S.: `dbs3_full_global.sh` is already using parallel dumps and logic can be copied from there.
[1] https://github.com/dmwm/CMSMonitoring/blob/master/sqoop/scripts/rucio_table_dumps.sh#L49
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.