scrapy / scrapy/form2request

Bug: form2request ignores submit buttons without attribute

Open Beginner friendly
#12 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5
Forks
5
PR merge metrics
No merged PRs in 30d

Description

Bug: form2request ignores submit buttons without value attribute

Summary

form2request does not include submit button data in the form submission when the button has a name attribute but no value attribute. The value None from button.get("value") is filtered out by the if v is not None check in the _data function.

Steps to Reproduce

from scrapy.http import HtmlResponse
from form2request import form2request

html = b'''<form name="login_form" action="/login" method="post">
<input type="text" name="username" />
<input type="password" name="password" />
<button type="submit" name="dispatch[auth.login]">Login</button>
</form>'''

response = HtmlResponse(
    url='https://example.com/login',
    body=html,
    encoding='utf-8'
)

form = response.xpath('//form[@name="login_form"]')[0]
request_data = form2request(form, {'username': 'test', 'password': 'pass'}, method='POST')

print(request_data.body.decode())
# Output: username=test&password=pass
# Expected: username=test&password=pass&dispatch[auth.login]=

Expected Behavior

The submit button data should be included in the form submission, even when the button has no value attribute. An empty value should be sent.

Expected output:

username=test&password=pass&dispatch[auth.login]=

Actual Behavior

Submit buttons without a value attribute are completely ignored and not included in the form submission.

Actual output:

username=test&password=pass

Environment

  • Python: 3.14
  • form2request: 0.2.0
  • OS: Linux

Root Cause

In form2request/_base.py, line 243:

values.extend((k, v) for k, v in items if v is not None)

When click_element.get("value") returns None (no value attribute), the field is filtered out.

Workaround

Add a value attribute to submit buttons in HTML:

<button type="submit" name="dispatch[auth.login]" value="">Login</button>

Or use manual urlencode instead of form2request when buttons don't have value attributes.

Impact

This affects forms that use submit buttons without value attributes to signal which action should be performed on the server side (common in PHP applications with dispatch[action_name] button naming pattern).

Comparison with scrapy.FormRequest

For reference, scrapy.http.FormRequest.from_response() correctly handles this case by including submit buttons with empty values:

# FormRequest.from_response() output:
# b'dispatch[auth.login]=&username=test&password=pass'

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in form2request/_base.py at line 243 and reproduce the issue with the HTML and request data shown in the report. Verify that a named submit button without a value attribute is included with an empty value, producing the expected request body.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
1/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
85/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.