elastic / elastic/semantic-code-search-indexer
Refactor parser.ts: move language-specific logic into LanguageConfiguration hooks
- Dominant language
- TypeScript
- Stars
- 19
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
`src/utils/parser.ts` has accumulated language-specific logic that doesn't belong there. The parser should be a generic engine that languages plug into, but instead it has:
1. **A growing `if/else` dispatch chain** (lines 381-409) — every new `parser: null` language requires a new branch. The `else` fallback silently produces zero chunks with no warning.
2. **Bash-specific import path handling** (lines 661-668) — hardcoded `if (langConfig.name === 'bash')` to treat all imports as file-type and normalize non-absolute paths.
3. **Python `__all__` parsing** (lines 700-742) — 40+ lines of Python-specific AST traversal embedded in the generic tree-sitter parser path.
4. **Python export filtering** (lines 795-799) — `if (langConfig.name === 'python')` to skip exports not in `__all__`.
5. **Bash export filtering** (lines 803-835) — 30+ lines of `if (langConfig.name === 'bash')` walking the AST to distinguish `export` from `readonly`/`local`/`declare`.
6. **JS/TS default export identifier resolution** (lines 758-767) — traverses parent nodes to find identifiers for `export default` statements.
That's ~100 lines of language-specific logic in what should be a generic parser. Every new language with non-trivial import/export semantics will add more `if (langConfig.name === '...')` branches.
## Proposal
Two changes to `LanguageConfiguration` in `src/utils/parser.ts`:
### 1. Declarative `parserType` for custom-parser dispatch
Replace the `if/else` chain with a `parserType` field:
```typescript
interface LanguageConfiguration {
// ... existing fields ...
parserType?: 'tree-sitter' | 'line-based' | 'paragraph' | 'whole-file' | 'delimiter';
delimiterPattern?: string; // used with 'delimiter' type
}
```
Current mapping:
- markdown → `delimiter` (with `delimiterPattern`)
- yaml, json → `line-based`
- handlebars → `whole-file`
- text, gradle → `paragraph`
- all tree-sitter languages → `tree-sitter` (inferred from `parser !== null`)
The silent `else { chunks = [] }` becomes a logged warning for unrecognized `parserType`.
### 2. `resolveImport` and `resolveExport` hooks
Add two optional hooks aligned to the existing capture name pipelines (`import.*` and `export.*`):
```typescript
interface LanguageConfiguration {
// ... existing fields ...
/** Post-process import captures. Default: resolve relative paths, classify module vs file. */
resolveImport?: (importPath: string, captures: ImportCaptures, context: ResolveContext) => ResolvedImport | null;
/** Post-process export captures. Default: return as-is. */
resolveExport?: (exportInfo: RawExportInfo, match: ExportMatch, context: ResolveContext) => ExportInfo | null;
}
```
- The **default `resolveImport`** handles the common case (relative paths → file type, everything else → module type). Bash overrides to treat all imports as file-type.
- The **default `resolveExport`** returns captures as-is. Python overrides to filter through `__all__`, Bash overrides to walk `declaration_command` nodes, JS/TS overrides to resolve default export identifiers.
- The hook set is **fixed and small** — tied to the capture pipelines, not an open-ended plugin system. Adding a new hook is a deliberate architecture decision.
### What moves where
| Current location in `parser.ts` | Moves to |
|----------------------------------|----------|
| Bash import normalization (lines 661-668) | `src/languages/bash.ts` → `resolveImport` |
| Python `__all__` parsing (lines 700-742) | `src/languages/python.ts` → `resolveExport` |
| Python export filtering (lines 795-799) | `src/languages/python.ts` → `resolveExport` |
| Bash export filtering (lines 803-835) | `src/languages/bash.ts` → `resolveExport` |
| JS/TS default export resolution (lines 758-767) | `src/languages/typescript.ts` / `javascript.ts` → `resolveExport` |
| `if/else` dispatch chain (lines 381-409) | Replaced by `switch` on `parserType` |
### What stays in `parser.ts`
- The generic tree-sitter parsing pipeline (query execution, chunk creation, semantic text preparation)
- Default `resolveImport` behavior (relative path resolution, module/file classification)
- Default `resolveExport` behavior (passthrough)
- The `parserType` switch for custom parsers
## Why this matters
Adding new languages currently requires editing `parser.ts` — a file that should be stable infrastructure. This refactor means:
- New tree-sitter languages: just provide queries in your config (same as today for simple languages like HCL or Go)
- New custom-parser languages: set `parserType` in your config, done — no code changes to `parser.ts`
- Languages with quirky import/export semantics: provide `resolveImport`/`resolveExport` in your config file
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.