firecrawl / firecrawl/cli

firecrawl CLI fails with HTTP 404 through HTTP proxy (bundled axios < 1.16.1, no CONNECT tunneling for HTTPS)

Open Beginner friendly
#172 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
631
Forks
100
Avg merge
5h 9m
Merged PRs (30d)
25

Description

## Summary

`firecrawl` CLI returns `Error: Request failed with status code 404` for **every** `scrape` / `search` / `map` command when run behind an HTTP proxy (e.g. `https_proxy=http://127.0.0.1:10808`). Root cause: the bundled `axios` is pinned at `1.15.2`, which predates the fix in [axios PR #10858](https://github.com/axios/axios/pull/10858) (shipped in `axios@1.16.1`). Pre-1.16.1 axios does **not** send a `CONNECT` request to tunnel HTTPS through an HTTP proxy; instead it forwards the request in forward-proxy style (`POST https://api.firecrawl.dev/v2/scrape HTTP/1.1`), which many HTTP proxies (xray, squid, etc.) reject.

## Environment

- `firecrawl-cli`: 1.19.27 (latest at time of reporting)
- firecrawl SDK dep: `axios` `1.15.2` (exact pin, no caret) — verified in `node_modules/firecrawl-cli/.../firecrawl/package.json` and the installed `axios@1.15.2` adapter source
- Node.js: v24.18.0
- OS: NixOS
- Proxy: xray-core `mixed` inbound on `127.0.0.1:10808` (HTTP + SOCKS5 on the same port)

## Reproduction

```bash
export https_proxy=http://127.0.0.1:10808
export http_proxy=http://127.0.0.1:10808
firecrawl scrape "https://example.com" --json
# => Error: Request failed with status code 404

firecrawl search "test" --limit 1
# => Error: Request failed with status code 404
```

Workaround (confirms the proxy is the trigger):
```bash
no_proxy="api.firecrawl.dev" firecrawl scrape "https://example.com" --json
# => 200 OK, returns markdown
```

## Expected vs actual

- **Expected:** axios opens a TLS tunnel through the proxy via `CONNECT api.firecrawl.dev:443`, then sends the HTTPS request inside the tunnel. (This is what `curl` does through the same proxy and it returns 200.)
- **Actual:** axios sends `POST https://api.firecrawl.dev/v2/scrape HTTP/1.1` directly to the proxy with the full URL as the request path. xray's mixed inbound has no route matching an HTTPS URL path and returns `404 Not Found`.

Captured request that axios actually sends to the proxy (plaintext, before TLS):
```
POST https://api.firecrawl.dev/v2/scrape HTTP/1.1
Host: api.firecrawl.dev
Authorization: Bearer fc-…
Content-Type: application/json
User-Agent: axios/1.15.2

{"url":"https://example.com","formats":["markdown"],"integration":"cli","onlyMainContent":false,"origin":"js-sdk@4.24.0"}
```

For comparison, `curl` through the same proxy correctly issues:
```
CONNECT api.firecrawl.dev:443 HTTP/1.1
```
and gets `200`.

## Root cause

This is a long-standing axios bug, not a firecrawl bug. It is tracked in:

- [axios#6320](https://github.com/axios/axios/issues/6320) — labeled `issue::security` ("Axios sends HTTPS data in cleartext to a proxy (regression)")
- [axios#6330](https://github.com/axios/axios/issues/6330) — "Using HTTP proxy for HTTPS address does not start a HTTP Connection"
- [axios#4531](https://github.com/axios/axios/issues/4531), [axios#3384](https://github.com/axios/axios/issues/3384) — older duplicates

Fixed by [axios PR #10858](https://github.com/axios/axios/pull/10858) (merged 2026-05-06 into `v1.x`), which integrates `https-proxy-agent` so axios performs proper `CONNECT` tunneling for HTTPS targets. The fix first shipped in **`axios@1.16.1`** (2026-05-13), per the axios `CHANGELOG.md` Security Fixes section.

Code-level confirmation in the installed `axios@1.15.2`:
- `lib/adapters/http.js` sets `options.path = location;` (the full target URL) for proxied HTTPS requests
- No `https-proxy-agent` / `HttpsProxyAgent` / `CONNECT` references anywhere in the adapter

## Suggested fix

Bump the firecrawl SDK's `axios` dependency from `1.15.2` to `>= 1.16.1`. The pin is currently exact (`"axios": "1.15.2"`), so it will not float on a reinstall. After the bump, axios will use `CONNECT` tunneling for HTTPS through HTTP proxies and the 404 disappears without any user-side workaround.

If a release is not imminent, an alternative is to document `no_proxy=api.firecrawl.dev` as a workaround in the CLI skill docs (since `api.firecrawl.dev` is a global CDN and typically does not need to go through a local proxy anyway).

## Notes

- This affects any user behind a corporate or local HTTP proxy that expects `CONNECT` for HTTPS (squid, tinyproxy, xray, mitmproxy, etc.), not just xray.
- SOCKS5 does **not** work around it either: setting `https_proxy=socks5h://...` makes axios fail with `protocol mismatch`, because the bundled axios has no SOCKS support.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the firecrawl SDK dependency declaration in package.json and confirm the exact axios 1.15.2 pin. Update it to at least 1.16.1, then run the documented scrape or search commands with https_proxy enabled. Done means HTTPS requests use proxy tunneling and no longer return the reported 404.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
cli, networking
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.