dmwm / dmwm/CMSMonitoring

Improve sqoop rucio table dumps with dynamic num-mappers

Open
#221 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
5
Forks
16
PR merge metrics
No merged PRs in 30d

Description

We are using sqoop `--num-mappers=40`[1] for all Rucio table dump queries. However, some tables like `rses, subscriptions` are so small tables and their run time can be decreased with `--num-mappers=1`.
Previous experiences showed that we can run 2 `--num-mappers-40` parallel, we did not implement it for Rucio table dumps since we don't have a time pressure. However, we can expect that our users will come with more Rucio table dump request, so we can implement it now.

So, my suggestions are:
- Run, at most, 2 Rucio table dumps parallel.
- Set --num-mappers=1 for "rses" and "subscriptions" tables.

P.S.: `dbs3_full_global.sh` is already using parallel dumps and logic can be copied from there.

[1] https://github.com/dmwm/CMSMonitoring/blob/master/sqoop/scripts/rucio_table_dumps.sh#L49

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.