No way to exclude folders from first-party detection for import sorting?

In my use-case, I want to apply import sorting to a large repository which has packages and also many stand-alone scripts. Those scripts import from their sibling and child python files, so I they should have a large amount of first party imports. Those scripts don't support relative imports (like `from .app import func`), and I don't want to maintain a large list of `known-first-party` imports in my Ruff configuration.

One way around this was to include every directory in `src`, just so long as there are no collisions with directories named the same as third-party packages. But there is no way to subtract from this list. I guess this is because the exclude list is for `.py` files but the `src` list is a list of directories. I tried using the `exclude` list, `force-exclude`, and also by honouring `.gitignore`.

Perhaps the exclude list should also affect the first-party detection? Or does anybody have any suggestions?

Directory structure:
```
ruff_test/
├─ main.py
├─ ruff.toml
└─ .mypy_cache/
└─ 3.8/
└─ numpy/
└─ ... only directories of JSON files
```

Before formatting:

```python
# main.py
from __future__ import annotations
import os
import pandas as pd
import numpy as np
```

```toml
# ruff.toml
src = [".", "**"]

[lint]
select = [
"I001", # unsorted-imports
]
```

After formatting with `ruff check --fix -v .`:

```python
# main.py
from __future__ import annotations

import os

import pandas as pd

import numpy as np
```

Verbose output:
```
[2023-12-14][15:39:16][ruff_cli::resolve][DEBUG] Using configuration file (via parent) at: ruff_test/ruff.toml
[2023-12-14][15:39:16][ruff_workspace::pyproject][DEBUG] `project.requires_python` in `pyproject.toml` will not be used to set `target_version` when using `ruff.toml`.
[2023-12-14][15:39:16][ruff_workspace::pyproject][DEBUG] `project.requires_python` in `pyproject.toml` will not be used to set `target_version` when using `ruff.toml`.
[2023-12-14][15:39:16][ruff_workspace::pyproject][DEBUG] `project.requires_python` in `pyproject.toml` will not be used to set `target_version` when using `ruff.toml`.
[2023-12-14][15:39:16][ruff_workspace::resolver][DEBUG] Ignored path via `exclude`: "ruff_test/.mypy_cache"
[2023-12-14][15:39:16][ruff_workspace::resolver][DEBUG] Included path via `include`: "ruff_test/main.py"
[2023-12-14][15:39:16][ruff_cli::commands::check][DEBUG] Identified files to lint in: 3.69786ms
[2023-12-14][15:39:16][ruff_cli::diagnostics][DEBUG] Checking: ruff_test/main.py
[2023-12-14][15:39:16][ruff_linter::rules::isort::categorize][DEBUG] Categorized 'pandas' as Known(ThirdParty) (NoMatch)
[2023-12-14][15:39:16][ruff_linter::rules::isort::categorize][DEBUG] Categorized 'os' as Known(StandardLibrary) (KnownStandardLibrary)
[2023-12-14][15:39:16][ruff_linter::rules::isort::categorize][DEBUG] Categorized 'numpy' as Known(FirstParty) (SourceMatch("ruff_test/.mypy_cache/3.8"))
[2023-12-14][15:39:16][ruff_linter::rules::isort::categorize][DEBUG] Categorized '__future__' as Known(Future) (Future)
[2023-12-14][15:39:16][ruff_linter::rules::isort::categorize][DEBUG] Categorized 'pandas' as Known(ThirdParty) (NoMatch)
[2023-12-14][15:39:16][ruff_linter::rules::isort::categorize][DEBUG] Categorized 'os' as Known(StandardLibrary) (KnownStandardLibrary)
[2023-12-14][15:39:16][ruff_linter::rules::isort::categorize][DEBUG] Categorized 'numpy' as Known(FirstParty) (SourceMatch("ruff_test/.mypy_cache/3.8"))
[2023-12-14][15:39:16][ruff_linter::rules::isort::categorize][DEBUG] Categorized '__future__' as Known(Future) (Future)
[2023-12-14][15:39:16][ruff_cli::commands::check][DEBUG] Checked 1 files in: 1.587502ms
Found 1 error (1 fixed, 0 remaining).
```

As you can see, it now thinks numpy is a first-party package because it matched `ruff_test/.mypy_cache/3.8/numpy/`

Start with the reproduction in ruff.toml and main.py, then run `ruff check --fix -v .` to observe how `src`, `exclude`, and `force-exclude` affect import categorization. Determine the intended interaction between excluded directories and first-party detection, and verify that numpy remains third-party while sibling project imports can still be recognized as first-party.

Written by the indexing model from the issue text.