fsspec / fsspec/filesystem_spec

Reading data from `raw.githubusercontent.com` hangs

Open
#1,471 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

What

When using the GitHub fsspec implementation, an attempt to open(...) data at raw.githubusercontent.com hangs indefinitely on my local machine.

Why

My DNS server returns both IPv4 and IPv6 addresses for raw.githubusercontent.com:

> nslookup raw.githubusercontent.com <IP_OF_MY_DNS_SERVER>

...

Name:	raw.githubusercontent.com
Address: 185.199.109.133
Name:	raw.githubusercontent.com
Address: 185.199.108.133
Name:	raw.githubusercontent.com
Address: 185.199.111.133
Name:	raw.githubusercontent.com
Address: 185.199.110.133
Name:	raw.githubusercontent.com
Address: 2606:50c0:8002::154
Name:	raw.githubusercontent.com
Address: 2606:50c0:8000::154
Name:	raw.githubusercontent.com
Address: 2606:50c0:8003::154
Name:	raw.githubusercontent.com
Address: 2606:50c0:8001::154

(Note: urllib3/requests gives an identical response for me.)

The IPv4 addresses are fine. However, I'm not able to establish a connection with the IPv6 address eg

❯ ping -w 30 2606:50c0:8001::154
PING 2606:50c0:8001::154(2606:50c0:8001::154) 56 data bytes

--- 2606:50c0:8001::154 ping statistics ---
30 packets transmitted, 0 received, 100% packet loss, time 29673ms

This creates a problem when a program attempts to first connect to an IPv6 address rather than IPv4 (ie program hangs waiting to establish a connection). Some (likely unresolved) issues from GitHub community that appear relevant:
Issue 1
Issue 2

Proposed fix

Introduce a timeout keyword on the filesystem instance here. This will allow a program to move forward in the event no response. This is also the recommended approach from requests. If this sounds okay, let me know and I would be happy to open a small PR.

Thanks for a cool project!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in fsspec/implementations/github.py at the linked filesystem implementation and inspect how opening raw.githubusercontent.com creates its connection. Review the requests timeout guidance cited in the issue and determine how the filesystem-level keyword should reach that request. Done means an IPv6 connection that receives no response no longer hangs indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, python
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.