scrapy / scrapy/cssselect

cssselect can't work on firefox

Open
#27 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
310
Forks
70
Avg merge
2h 9m
Merged PRs (30d)
4

Description

Firefox is unlike other xpath implementations in that name() returns an upper-cased string. cssselect's translation of the nth-child selector (for example) uses "name() = 'foo'" which will never possibly match, due to the above oddity. One workaround is to set HTMLTranslator.lower_case_element_names to False, and write selectors like 'LI:nth-child(2)', which will result in a working xpath for firefox, but won't work on any other xpath implementation.

I see two possible solutions:

  1. Call lower-case() wherever name() is called.
  2. Factor out the use of name() entirely, replacing [name() = 'foo'] with [self::foo].

Demonstration of the problem and solution here: (the contrast between chrome and firefox is stark, ie is like chrome)
http://fiddle.jshell.net/J7VrG/10/show/light/

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting HTMLTranslator.lower_case_element_names and the nth-child translation that emits name() comparisons. Reproduce the generated XPath in Firefox and another XPath implementation using the linked demonstration. Done means nth-child selectors work consistently across the implementations without requiring different selector casing.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.