Very slow scrolling when using multiple tabs and splits

## Information

**VIM version**

VIM - Vi IMproved 8.1 (2018 May 18, compiled Nov 09 2019 12:59:40)
Included patches: 1-2269

Operating System:

PRETTY_NAME="Debian GNU/Linux bullseye/sid"
NAME="Debian GNU/Linux"

## What went wrong

Editing https://github.com/Jeansen/bcrm/blob/master/bcrm.sh with multiple splits and tabs makes scrolling `` terribly slow. Disabling Ale `:ALEDisableBuffer` resolves it.

The problem actually persists for a while already and not only since the latest ALE or VIM update but it is only now that I found some time to dig a bit deeper ....

## Reproducing the bug
0. Make sure ALE is enabled `:ALEEnable` or `let g:ale_enabled = 1`
1. Open up https://github.com/Jeansen/bcrm/blob/master/bcrm.sh in vim
2. Create a split `C-w v`
3. Create a tab `:tab split`
4. Create a split `C-w v`
(There should be two tabs now with two buffers (one split) each)
5. Go back to the previous tab `:tabprevious`
6. Scroll in one of the splits of the first tab `C-e`

### :ALEInfo
```
Current Filetype: sh
Available Linters: ['language_server', 'shell', 'shellcheck']
Enabled Linters: ['language_server', 'shellcheck']
Suggested Fixers:
'remove_trailing_lines' - Remove all blank lines at the end of a file.
'shfmt' - Fix sh files with shfmt.
'trim_whitespace' - Remove all trailing whitespace characters at the end of every line.
Linter Variables:

let g:ale_sh_language_server_executable = 'bash-language-server'
let g:ale_sh_language_server_use_global = 0
let g:ale_sh_shellcheck_change_directory = 1
let g:ale_sh_shellcheck_dialect = 'auto'
let g:ale_sh_shellcheck_exclusions = ''
let g:ale_sh_shellcheck_executable = 'shellcheck'
let g:ale_sh_shellcheck_options = ''
Global Variables:

let g:ale_cache_executable_check_failures = v:null
let g:ale_change_sign_column_color = 0
let g:ale_command_wrapper = ''
let g:ale_completion_delay = v:null
let g:ale_completion_enabled = 0
let g:ale_completion_max_suggestions = v:null
let g:ale_echo_cursor = 1
let g:ale_echo_msg_error_str = 'Error'
let g:ale_echo_msg_format = '%s'
let g:ale_echo_msg_info_str = 'Info'
let g:ale_echo_msg_warning_str = 'Warning'
let g:ale_enabled = 1
let g:ale_fix_on_save = 1
let g:ale_fixers = {'javascript': ['eslint']}
let g:ale_history_enabled = 1
let g:ale_history_log_output = 1
let g:ale_keep_list_window_open = 0
let g:ale_lint_delay = 0
let g:ale_lint_on_enter = 1
let g:ale_lint_on_filetype_changed = 1
let g:ale_lint_on_insert_leave = 1
let g:ale_lint_on_save = 1
let g:ale_lint_on_text_changed = 'normal'
let g:ale_linter_aliases = {}
let g:ale_linters = {'markdown': ['remark'], 'sh': ['shellcheck', 'language_server'], 'java': ['javalsp'], 'javascript': ['eslint']}
let g:ale_linters_explicit = 0
let g:ale_list_vertical = 0
let g:ale_list_window_size = 5
let g:ale_loclist_msg_format = '%s'
let g:ale_lsp_root = {}
let g:ale_max_buffer_history_size = 20
let g:ale_max_signs = -1
let g:ale_maximum_file_size = v:null
let g:ale_open_list = 0
let g:ale_pattern_options = v:null
let g:ale_pattern_options_enabled = v:null
let g:ale_set_balloons = 0
let g:ale_set_highlights = 1
let g:ale_set_loclist = 0
let g:ale_set_quickfix = 1
let g:ale_set_signs = 1
let g:ale_sign_column_always = 1
let g:ale_sign_error = '┃'
let g:ale_sign_info = '┃'
let g:ale_sign_offset = 1000000
let g:ale_sign_style_error = '┃'
let g:ale_sign_style_warning = '┃'
let g:ale_sign_warning = '┃'
let g:ale_sign_highlight_linenrs = 0
let g:ale_statusline_format = ['✗ %d', '⚠ %d', '✓ ok']
let g:ale_type_map = {}
let g:ale_use_global_executables = v:null
let g:ale_virtualtext_cursor = 0
let g:ale_warn_about_trailing_blank_lines = 1
let g:ale_warn_about_trailing_whitespace = 1
Command History:

(executable check - success) /home/marcel/node_modules/.bin/bash-language-server
(executable check - success) shellcheck
(finished - exit code 0) ['bash', '-c', '''shellcheck'' --version']
```