imrpvements

# Creating a Sandbox for Testing Against Real Files

The simplest approach is a **fixture-based test harness** — no external sandbox tooling needed. You're testing a pure TypeScript library, so you can run tests directly against real Markdown files stored in your repo.

## Fixture-Based Test Harness (Recommended)

**1. Create a fixtures directory:**
```
tests/fixtures/
├── sample-article.md
├── nested-lists.md
├── code-heavy.md
├── html-mixed.md
└── edge-cases.md
```

**2. Write a test that loads each fixture and runs `extract()`:**
```typescript
// tests/fixtures.test.ts
import { extract, REGEXP_FENCED_CODE, REGEXP_INLINE_CODE } from '../src';
import * as fs from 'fs';
import * as path from 'path';

const fixturesDir = path.join(__dirname, 'fixtures');
const fixtures = fs.readdirSync(fixturesDir).filter(f => f.endsWith('.md'));

describe('extract() against real Markdown files', () => {
fixtures.forEach(file => {
it(`extracts structured data from ${file}`, () => {
const md = fs.readFileSync(path.join(fixturesDir, file), 'utf-8');
const result = extract(md);

// Snapshot the result to catch regressions
expect(result).toMatchSnapshot();

// Basic sanity assertions
expect(result).toHaveProperty('headers');
expect(result).toHaveProperty('links');
expect(result).toHaveProperty('codeBlocks');
expect(result).toHaveProperty('inlineCode');
expect(result).toHaveProperty('html');
});
});
});
```

**3. Add specific assertion tests for the new regexes:**
```typescript
// tests/code-and-html.test.ts
import { extract, REGEXP_FENCED_CODE, REGEXP_INLINE_CODE, REGEXP_HTML } from '../src';

describe('REGEXP_FENCED_CODE', () => {
it('matches triple-backtick blocks', () => {
const md = 'text\n```js\nconst x = 1;\n```\nmore';
const match = md.match(REGEXP_FENCED_CODE);
expect(match).not.toBeNull();
expect(match![0]).toContain('const x = 1;');
});

it('matches tilde-fenced blocks', () => {
const md = 'text\n~~~python\nprint("hi")\n~~~\nmore';
const match = md.match(REGEXP_FENCED_CODE);
expect(match).not.toBeNull();
});

it('does not match inline code', () => {
const md = 'Use `foo` here.';
const match = md.match(REGEXP_FENCED_CODE);
expect(match).toBeNull();
});
});

describe('REGEXP_INLINE_CODE', () => {
it('matches single-backtick spans', () => {
const md = 'Use `foo` here.';
const match = md.match(REGEXP_INLINE_CODE);
expect(match).not.toBeNull();
expect(match![0]).toBe('`foo`');
});

it('does not match fenced blocks', () => {
const md = 'text\n```\ncode\n```';
const match = md.match(REGEXP_INLINE_CODE);
expect(match).toBeNull();
});
});

describe('extract()', () => {
it('returns all extracted fields', () => {
const md = `# Title\n\n[link](https://example.com)\n\n\`\`\`js\ncode\n\`\`\`\n\nInline \`code\` here.\n

`;
const result = extract(md);
expect(result.headers).toHaveLength(1);
expect(result.links).toHaveLength(1);
expect(result.codeBlocks).toHaveLength(1);
expect(result.inlineCode).toHaveLength(1);
expect(result.html).toHaveLength(1);
});
});
```

This runs with your existing Jest setup — no Docker, no VM, no external service.

## Optional: True Isolation Sandbox

If you want stronger isolation (e.g., for running untrusted test inputs), you can use a container-based sandbox tool. `onbox` is one option that mounts a host directory and runs commands in an isolated Node image:

```bash
# Mount your project, run tests inside
onbox create --image node:22 --mount ~/dev/markdown-regex:/workspace --connect
# Then inside:
cd /workspace && npm install && npm test
```

But for a regex library, **fixture-based tests in your existing Jest suite are sufficient and simpler**.

---

# Adding the Three New Regexes + `extract()`

## 1. `REGEXP_FENCED_CODE`

Matches triple-backtick or triple-tilde fences, with optional language tag.

```typescript
// src/index.ts

/** Matches fenced code blocks: ```lang\ncode\n``` or ~~~lang\ncode\n~~~ */
export const REGEXP_FENCED_CODE = /(?:^|\n)(`{3,}|~{3,})([\w-]*)\r?\n([\s\S]*?)\r?\n\1(?=\n|$)/g;
```

**Capture groups:**
- `1` — fence marker (backticks or tildes)
- `2` — language identifier (may be empty)
- `3` — code content

**Notes:**
- Uses lazy `[\s\S]*?` to stop at the first matching fence
- `{3,}` allows longer fences (CommonMark spec)
- The `g` flag is required for `matchAll`

## 2. `REGEXP_INLINE_CODE`

Matches `` `code` `` spans (one or more backticks), without crossing blank lines.

```typescript
/** Matches inline code spans: `code` */
export const REGEXP_INLINE_CODE = /(?, ,
*/
export const REGEXP_HTML = /<\/?[a-z][\w-]*(?:\s+[\w-]+(?:=(?:"[^"]*"|'[^']*'|[^\s>]+))?)*\s*\/?>/gi;
```

**Notes:**
- `i` flag for case-insensitive tag names
- Handles attributes with double quotes, single quotes, or unquoted values
- Does **not** match comments (``) or doctypes — those would need separate patterns if needed

## 4. `extract(md)` Function

Ties everything together into a structured result.

```typescript
// src/extract.ts
import {
REGEXP_HEADER,
REGEXP_LINK,
REGEXP_IMAGE,
REGEXP_STRONG,
REGEXP_ITALIC,
REGEXP_DEL,
REGEXP_BLOCKQUOTE,
REGEXP_HR,
REGEXP_UL_LIST,
REGEXP_OL_LIST,
REGEXP_FENCED_CODE,
REGEXP_INLINE_CODE,
REGEXP_HTML,
} from './index';

export interface ExtractResult {
headers: { level: number; text: string }[];
links: { text: string; url: string }[];
images: { alt: string; url: string }[];
bold: string[];
italic: string[];
strikethrough: string[];
blockquotes: string[];
horizontalRules: string[];
unorderedLists: string[];
orderedLists: string[];
codeBlocks: { language: string; code: string }[];
inlineCode: string[];
html: string[];
}

/**
* Extracts structured data from a Markdown string using the library's regex constants.
*
* IMPORTANT: Fenced code blocks are extracted FIRST and removed from the working string
* so that other patterns (links, bold, etc.) do not match syntax inside code.
*/
export function extract(markdown: string): ExtractResult {
const result: ExtractResult = {
headers: [],
links: [],
images: [],
bold: [],
italic: [],
strikethrough: [],
blockquotes: [],
horizontalRules: [],
unorderedLists: [],
orderedLists: [],
codeBlocks: [],
inlineCode: [],
html: [],
};

// --- Step 1: Extract fenced code blocks and remove them from the source ---
let working = markdown;

const fencedMatches = [...working.matchAll(REGEXP_FENCED_CODE)];
for (const m of fencedMatches) {
result.codeBlocks.push({
language: m[2] || '',
code: m[3],
});
}
// Replace fenced blocks with a placeholder so later patterns don't match inside them
working = working.replace(REGEXP_FENCED_CODE, '\n__CODE_BLOCK__\n');

// --- Step 2: Extract inline code and remove it ---
const inlineMatches = [...working.matchAll(REGEXP_INLINE_CODE)];
for (const m of inlineMatches) {
result.inlineCode.push(m[2]);
}
working = working.replace(REGEXP_INLINE_CODE, '__INLINE_CODE__');

// --- Step 3: Run all other patterns against the cleaned string ---

// Headers: capture level from the `#` count
for (const m of working.matchAll(REGEXP_HEADER)) {
const hashes = m[1]; // assumes REGEXP_HEADER has a capture group for the # prefix
result.headers.push({
level: hashes.length,
text: m[2].trim(),
});
}

// Links
for (const m of working.matchAll(REGEXP_LINK)) {
result.links.push({ text: m[1], url: m[2] });
}

// Images
for (const m of working.matchAll(REGEXP_IMAGE)) {
result.images.push({ alt: m[1], url: m[2] });
}

// Bold
for (const m of working.matchAll(REGEXP_STRONG)) {
result.bold.push(m[1]);
}

// Italic
for (const m of working.matchAll(REGEXP_ITALIC)) {
result.italic.push(m[1]);
}

// Strikethrough
for (const m of working.matchAll(REGEXP_DEL)) {
result.strikethrough.push(m[1]);
}

// Blockquotes
for (const m of working.matchAll(REGEXP_BLOCKQUOTE)) {
result.blockquotes.push(m[1]);
}

// Horizontal rules
for (const m of working.matchAll(REGEXP_HR)) {
result.horizontalRules.push(m[0]);
}

// Unordered lists
for (const m of working.matchAll(REGEXP_UL_LIST)) {
result.unorderedLists.push(m[1]);
}

// Ordered lists
for (const m of working.matchAll(REGEXP_OL_LIST)) {
result.orderedLists.push(m[1]);
}

// Raw HTML
for (const m of working.matchAll(REGEXP_HTML)) {
result.html.push(m[0]);
}

return result;
}
```

**Critical ordering detail:** `extract()` runs fenced-code extraction **first**, then inline code, then everything else. This prevents false positives like `**bold**` inside a code block being counted as bold text — a known limitation of regex-only Markdown parsing.

## 5. Export Everything

```typescript
// src/index.ts (add to existing exports)
export { REGEXP_FENCED_CODE } from './code';
export { REGEXP_INLINE_CODE } from './code';
export { REGEXP_HTML } from './html';
export { extract } from './extract';
export type { ExtractResult } from './extract';
```

---

## Summary

| Task | Approach |
|------|----------|
| **Sandbox for tests** | Jest + `tests/fixtures/*.md` files. No external sandbox needed. |
| **`REGEXP_FENCED_CODE`** | `/(?:^|\n)(`{3,}|~{3,})([\w-]*)\r?\n([\s\S]*?)\r?\n\1(?=\n|$)/g` |
| **`REGEXP_INLINE_CODE`** | `/(?]+))?)*\s*\/?>/gi` |
| **`extract(md)`** | Extract fenced → inline → rest. Return typed `ExtractResult`. |

The fixture harness is the highest-value addition — it catches regressions silently, requires zero infrastructure, and runs in your existing CI pipeline.

Start by inspecting the existing exports and regex definitions in src/index.ts, then review the Jest setup and current tests before adding tests/fixtures/*.md. The work is complete when the three new regexes and extract() are exported, fixture and focused tests pass, and the extracted fields match the specified result shape without matching syntax inside fenced code.

Written by the indexing model from the issue text.