Cannot index local hosted documentation server

Before submitting your bug report

• I believe this is a bug. I'll try to join the Continue Discord for questions
• I'm not able to find an open issue that reports the same bug
• I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS: Windows10
- Continue version: v1.0.2
- IDE version: VSCode 1.97.2
- Model: Ollama llama3.3 / qwen2.5-coder:7b
- config:
  
name: My Config
version: 1.0.0
schema: v1

models:

  - name: Ollama Chat (Remote)
    provider: ollama
    model: AUTODETECT
    roles:
      - chat
      - edit
      - apply
      - summarize
    apiBase: http://localhost:11434
    contextLength: 12288

  - name: Ollama Autocomplete (Remote)
    provider: ollama
    model: qwen2.5-coder:7b
    roles:
      - autocomplete
    apiBase: http://localhost:11434

  - name: Ollama Embeddings (Remote)
    provider: ollama
    model: nomic-embed-text
    roles:
      - embed
    apiBase: http://localhost:11434

rules:
  - Always give concise responses

context:
  - provider: codebase
  - provider: currentFile

docs:
  - name: MyDocuments
    startUrl: http://127.0.0.1:8091
    useLocalCrawling: true

Description

I would like Continue to index a locally hosted documentation site. When I add this server to config.yaml, I get the error popup "Failed to index http://127.0.0.1:8091". I see a GET issued to the http server from the Continue crawler, and I am able to access the server on a local browser, so I don't think it is an access issue.

To reproduce

1. Setup a local http server with html documentation at http://127.0.0.1:8091
2. Update config.yaml to specify http://127.0.0.1:8091 as a documentation source
3. Observe error message "Failed to index http://127.0.0.1:8091"

Log output

ERR [Extension Host] Default crawler failed, trying backup:  Error: Failed to crawl site (500): Error in Continue free trial server: Request failed with status code 400. Error: Bad Request  - [{"code":"custom","message":"URL must have a valid top-level domain or be a valid path","path":["url"]}]
	at DefaultCrawler.crawl (c:\Users\<user>\.vscode\extensions\continue.continue-1.0.2-win32-x64\out\extension.js:520300:17)
	at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
	at async DocsCrawler.crawl (c:\Users\<user>\.vscode\extensions\continue.continue-1.0.2-win32-x64\out\extension.js:524206:30)
	at async _DocsService.indexAndAdd (c:\Users\<user>\.vscode\extensions\continue.continue-1.0.2-win32-x64\out\extension.js:552622:29)
	at async Promise.allSettled (index 0)
	at async _DocsService.syncDocs (c:\Users\<user>\.vscode\extensions\continue.continue-1.0.2-win32-x64\out\extension.js:552970:11)
	at async _DocsService.handleConfigUpdate (c:\Users\<user>\.vscode\extensions\continue.continue-1.0.2-win32-x64\out\extension.js:552456:11)
error @ log.ts:456
console.ts:137 [Extension Host] Default crawler failed, trying backup:  Error: Failed to crawl site (500): Error in Continue free trial server: Request failed with status code 400. Error: Bad Request  - [{"code":"custom","message":"URL must have a valid top-level domain or be a valid path","path":["url"]}]
	at DefaultCrawler.crawl (c:\Users\<user>\.vscode\extensions\continue.continue-1.0.2-win32-x64\out\extension.js:520300:17)
	at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
	at async DocsCrawler.crawl (c:\Users\<user>\.vscode\extensions\continue.continue-1.0.2-win32-x64\out\extension.js:524206:30)
	at async _DocsService.indexAndAdd (c:\Users\<user>\.vscode\extensions\continue.continue-1.0.2-win32-x64\out\extension.js:552622:29)
	at async Promise.allSettled (index 0)
	at async _DocsService.syncDocs (c:\Users\<user>\.vscode\extensions\continue.continue-1.0.2-win32-x64\out\extension.js:552970:11)
	at async _DocsService.handleConfigUpdate (c:\Users\<user>\.vscode\extensions\continue.continue-1.0.2-win32-x64\out\extension.js:552456:11)
Ats @ console.ts:137
console.ts:137 [Extension Host] [CheerioCrawler] Starting crawl from: http://127.0.0.1:8091/ - Max Depth: 4
console.ts:137 [Extension Host] Crawl completed
log.ts:456   ERR [Extension Host] Docs indexing: Error getting readability for URL http://127.0.0.1:8091/
error @ log.ts:456
console.ts:137 [Extension Host] Docs indexing: Error getting readability for URL http://127.0.0.1:8091/
Ats @ console.ts:137
log.ts:456   ERR [Extension Host] No embeddings were created for site: http://127.0.0.1:8091
 Num chunks: 0
error @ log.ts:456
console.ts:137 [Extension Host] No embeddings were created for site: http://127.0.0.1:8091
 Num chunks: 0