`follow_if_*` options not working (always false)

Open

#252 opened on Aug 11, 2018 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty: 2/5
Estimated time: 1-3 hours
Newbie friendliness: 45/100

Issue type: Bug
Clarity: Clearly specified
Activity status: Stale
Tech stack: javascript, node.js
Domain: networking

Research direction

Start in lib/needle.js at Needle.prototype.should_follow and the send_request flow around the referenced lines. Run the supplied Node.js reproduction with follow_if_same_host enabled, then verify that same-host and same-protocol redirects are followed instead of stopping at the 302 response.

Written by the indexing model from the issue text.

Description

Needle.prototype.should_follow expects to recive original url as parsed value, but actually receives it as a string. This makes follow_if_same_host and follow_if_same_protocol always return false, since they are comparing on the string.

To reproduce, try setting follow_if_same_host to true and query some website that redirects to same host. I had problems with this one:

Code to reproduce

'use strict';

const needle = require('needle');

const _makeRequest = async () => {
  return needle('get', `http://www.parkflyer.ru/`, {
    compressed: true,
    accept: 'text/html',
    open_timeout: 10000,
    response_timeout: 10000,
    read_timeout: 10000,
    follow_max: 5,
    rejectUnauthorized: false,
    follow_if_same_host: true,
  });
}

_makeRequest()
  .then(({statusCode, body}) => console.log({statusCode}))
  .catch((err) => console.log({err}));

Output of the above

{ statusCode: 302 }

Things I printed from within `function matches(property)` inside `send_request`

{ property: 'host',
  original: 'http://www.parkflyer.ru/',
  location: 'http://www.parkflyer.ru/ru/',
  originalProperty: undefined }

Same site, but with `curl` following redirects

$ curl -LI http://www.parkflyer.ru/
HTTP/1.1 302 FOUND
Server: nginx
Date: Sat, 11 Aug 2018 08:58:05 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Keep-Alive: timeout=20
Vary: Cookie
Location: http://www.parkflyer.ru/ru/
Set-Cookie: sessionid=lsp455s7plixmol4yarfi1spu6j477va; expires=Sat, 25-Aug-2018 08:58:05 GMT; httponly; Max-Age=1209600; Path=/

HTTP/1.1 200 OK
Server: nginx
Date: Sat, 11 Aug 2018 08:58:05 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Keep-Alive: timeout=20
Vary: Cookie
Content-Language: ru
Set-Cookie: vtime="2018-08-11T11:58:05.718622"; Path=/
Set-Cookie: country_from_ip=RU; Path=/
Set-Cookie: referrer=; expires=Tue, 08-Aug-2028 08:58:05 GMT; Max-Age=315360000; Path=/
Set-Cookie: landing="http://www.parkflyer.ru/ru/"; Path=/
Set-Cookie: nv_event_id=None; expires=Tue, 08-Aug-2028 08:58:05 GMT; Max-Age=315360000; Path=/
Set-Cookie: currency=RUR; Path=/
Set-Cookie: sessionid=9b4vys6u4vfdlz5swrpr8jssxkz9npvv; expires=Sat, 25-Aug-2018 08:58:05 GMT; httponly; Max-Age=1209600; Path=/
Set-Cookie: refurl=; Path=/
Set-Cookie: csrftoken=Tnjfvk345BORT3zM1yCpKWzTDQg2LyGS; expires=Sat, 10-Aug-2019 08:58:05 GMT; Max-Age=31449600; Path=/
Set-Cookie: news=%23articles_all; Path=/
Set-Cookie: stuff_ive_seen=%7B%22product%22%3A%203127985%2C%20%22blog_entry_article%22%3A%2015834%2C%20%22blog_entry_vio%22%3A%2015830%2C%20%22comment_product%22%3A%2026790371%2C%20%22blog_entry_review%22%3A%200%2C%20%22blog_entry_news%22%3A%2015531%7D; expires=Sun, 11-Aug-2019 08:58:05 GMT; Max-Age=31536000; Path=/

This comparse as if original was parsed output of, say, url.parse, which has host and protocol properties (see line 409):

https://github.com/tomas/needle/blob/9374dcc03e4a2499ae08de77aff289a43918e70b/lib/needle.js#L404-L411

Here's an invokation which passes uri as is:

https://github.com/tomas/needle/blob/9374dcc03e4a2499ae08de77aff289a43918e70b/lib/needle.js#L494

uri is an argument to Needle.prototype.send_request:

https://github.com/tomas/needle/blob/9374dcc03e4a2499ae08de77aff289a43918e70b/lib/needle.js#L426

… which seems to always be a string

either from #start()
or previous redirects which is result of url.resolve which returns a String:
- https://github.com/tomas/needle/blob/9374dcc03e4a2499ae08de77aff289a43918e70b/lib/needle.js#L520
- https://github.com/tomas/needle/blob/9374dcc03e4a2499ae08de77aff289a43918e70b/lib/needle.js#L533

Dominant language: JavaScript
Stars: 1.6k
Forks: 237
PR merge metrics: No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Similar issues

[Duplicate Code] Repeated filtered model-alias assembly in proxy server Open

code-quality refactoring

Difficulty 2/5 1-3 hours Newbie friendliness 84/100

github/gh-aw-firewall#8816 · Opened Sep 20, 2026
quickjs: support `timeout=None` (no per-call deadline) in `CodeInterpreterMiddleware` Open

integration:quickjs org:external priority:backlog topic:code-interpreter topic:middleware type:feature

Difficulty 2/5 1-3 hours Newbie friendliness 74/100

langchain-ai/deepagents#6450 · Opened Sep 20, 2026
[optimization:agents-md-curator] AGENTS.md: compress duplicated Debugging & Logging section and merge redundant autopilot rules Open

optimization optimization:agents-md-curator

Difficulty 2/5 1-3 hours Newbie friendliness 86/100

githubnext/gh-aw-cao#13143 · Opened Sep 20, 2026
[BUG] schema-compat/zodToJsonSchema: nullable zod v4 enum/literal widens to type:[X,"null"] but keeps enum/const, so exported schema rejects null Open

status: needs triage

Difficulty 2/5 1-3 hours Newbie friendliness 88/100

mastra-ai/mastra#24516 · 2 comments · Opened Sep 20, 2026
Kubernetes request flow diagram 404s on both copies of the tutorial Open

Difficulty 2/5 1-3 hours Newbie friendliness 88/100

layer5io/docs#1263 · Opened Sep 20, 2026

More JavaScript issues