Vim extension settings persist in VSCode even after disabling / uninstalling Vim extension

Describe the bug
After installing Vim extension, I updated 2 settings in .vimrc:
set number
set relativenumber

in Vim.
These settings persist in VSCode even after I uninstall the Vim Extension in VSCode and it still
shows line numbers in relative mode (relative to where the cursor is placed in the file)

To Reproduce
Steps to reproduce the behavior:

1. Install VIM extension in VSCode
2. Update vim config settings by adding 'set number' and 'set relativenumber'
3. Check if these settings are being reflected in VSCode.
4. Disable/ Uninstall the VIM Extension from VSCode. Reload, close and re-open vs code.
5. Open a file and click on a random line. The linenumbers are still set to relative even after uninstalling vim extension.

Expected behavior
After disabling / uninstalling VSCode, the linenumbers should be set back to absolute.

Screenshots

Environment (please complete the following information):

• Extension (VsCodeVim) version: v1.18.5
• VSCode version: October 2020 (version 1.51)
• OS: Ubuntu 20.04

No file, test, or entry point is named. Start by reproducing the issue with VSCodeVim v1.18.5 on VSCode 1.51 and Ubuntu 20.04, then trace how the extension applies the .vimrc settings and what remains after disabling or uninstalling it. Done means line numbers return to absolute mode after the extension is removed.