elastic / elastic/semantic-code-search-indexer

Refactor parser.ts: move language-specific logic into LanguageConfiguration hooks

Open
#162 0 comments 0 reactions 1 assignee Claimed by @simianhacker View on GitHub
Dominant language
TypeScript
Stars
19
Forks
10
PR merge metrics
No merged PRs in 30d

Description

## Problem

`src/utils/parser.ts` has accumulated language-specific logic that doesn't belong there. The parser should be a generic engine that languages plug into, but instead it has:

1. **A growing `if/else` dispatch chain** (lines 381-409) — every new `parser: null` language requires a new branch. The `else` fallback silently produces zero chunks with no warning.

2. **Bash-specific import path handling** (lines 661-668) — hardcoded `if (langConfig.name === 'bash')` to treat all imports as file-type and normalize non-absolute paths.

3. **Python `__all__` parsing** (lines 700-742) — 40+ lines of Python-specific AST traversal embedded in the generic tree-sitter parser path.

4. **Python export filtering** (lines 795-799) — `if (langConfig.name === 'python')` to skip exports not in `__all__`.

5. **Bash export filtering** (lines 803-835) — 30+ lines of `if (langConfig.name === 'bash')` walking the AST to distinguish `export` from `readonly`/`local`/`declare`.

6. **JS/TS default export identifier resolution** (lines 758-767) — traverses parent nodes to find identifiers for `export default` statements.

That's ~100 lines of language-specific logic in what should be a generic parser. Every new language with non-trivial import/export semantics will add more `if (langConfig.name === '...')` branches.

## Proposal

Two changes to `LanguageConfiguration` in `src/utils/parser.ts`:

### 1. Declarative `parserType` for custom-parser dispatch

Replace the `if/else` chain with a `parserType` field:

```typescript
interface LanguageConfiguration {
// ... existing fields ...
parserType?: 'tree-sitter' | 'line-based' | 'paragraph' | 'whole-file' | 'delimiter';
delimiterPattern?: string; // used with 'delimiter' type
}
```

Current mapping:
- markdown → `delimiter` (with `delimiterPattern`)
- yaml, json → `line-based`
- handlebars → `whole-file`
- text, gradle → `paragraph`
- all tree-sitter languages → `tree-sitter` (inferred from `parser !== null`)

The silent `else { chunks = [] }` becomes a logged warning for unrecognized `parserType`.

### 2. `resolveImport` and `resolveExport` hooks

Add two optional hooks aligned to the existing capture name pipelines (`import.*` and `export.*`):

```typescript
interface LanguageConfiguration {
// ... existing fields ...

/** Post-process import captures. Default: resolve relative paths, classify module vs file. */
resolveImport?: (importPath: string, captures: ImportCaptures, context: ResolveContext) => ResolvedImport | null;

/** Post-process export captures. Default: return as-is. */
resolveExport?: (exportInfo: RawExportInfo, match: ExportMatch, context: ResolveContext) => ExportInfo | null;
}
```

- The **default `resolveImport`** handles the common case (relative paths → file type, everything else → module type). Bash overrides to treat all imports as file-type.
- The **default `resolveExport`** returns captures as-is. Python overrides to filter through `__all__`, Bash overrides to walk `declaration_command` nodes, JS/TS overrides to resolve default export identifiers.
- The hook set is **fixed and small** — tied to the capture pipelines, not an open-ended plugin system. Adding a new hook is a deliberate architecture decision.

### What moves where

| Current location in `parser.ts` | Moves to |
|----------------------------------|----------|
| Bash import normalization (lines 661-668) | `src/languages/bash.ts` → `resolveImport` |
| Python `__all__` parsing (lines 700-742) | `src/languages/python.ts` → `resolveExport` |
| Python export filtering (lines 795-799) | `src/languages/python.ts` → `resolveExport` |
| Bash export filtering (lines 803-835) | `src/languages/bash.ts` → `resolveExport` |
| JS/TS default export resolution (lines 758-767) | `src/languages/typescript.ts` / `javascript.ts` → `resolveExport` |
| `if/else` dispatch chain (lines 381-409) | Replaced by `switch` on `parserType` |

### What stays in `parser.ts`

- The generic tree-sitter parsing pipeline (query execution, chunk creation, semantic text preparation)
- Default `resolveImport` behavior (relative path resolution, module/file classification)
- Default `resolveExport` behavior (passthrough)
- The `parserType` switch for custom parsers

## Why this matters

Adding new languages currently requires editing `parser.ts` — a file that should be stable infrastructure. This refactor means:
- New tree-sitter languages: just provide queries in your config (same as today for simple languages like HCL or Go)
- New custom-parser languages: set `parserType` in your config, done — no code changes to `parser.ts`
- Languages with quirky import/export semantics: provide `resolveImport`/`resolveExport` in your config file

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.