OverflowError: string longer than 2147483647 bytes for large datasets
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 755
- Forks
- 128
- PR merge metrics
- No merged PRs in 30d
Description
Hi, I want to upload a large dataset 2.7GB and 4.6M features but I get the next error:
Traceback (most recent call last): File "publish_dataset.py", line 62, in <module> publish_dataset() File "publish_dataset.py", line 49, in publish_dataset openml_dataset.publish() File "C:\Users\20210595\.conda\envs\tableshift\lib\site-packages\openml\base.py", line 135, in publish response_text = openml._api_calls._perform_api_call( File "C:\Users\20210595\.conda\envs\tableshift\lib\site-packages\openml\_api_calls.py", line 118, in _perform_api_call response = _read_url_files(url, data=data, file_elements=file_elements) File "C:\Users\20210595\.conda\envs\tableshift\lib\site-packages\openml\_api_calls.py", line 325, in _read_url_files return _send_request( File "C:\Users\20210595\.conda\envs\tableshift\lib\site-packages\openml\_api_calls.py", line 383, in _send_request response = session.post(url, data=data, files=files, headers=_HEADERS) File "C:\Users\20210595\.conda\envs\tableshift\lib\site-packages\requests\sessions.py", line 637, in post return self.request("POST", url, data=data, json=json, **kwargs) File "C:\Users\20210595\.conda\envs\tableshift\lib\site-packages\requests\sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "C:\Users\20210595\.conda\envs\tableshift\lib\site-packages\requests\sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "C:\Users\20210595\.conda\envs\tableshift\lib\site-packages\requests\adapters.py", line 667, in send resp = conn.urlopen( File "C:\Users\20210595\.conda\envs\tableshift\lib\site-packages\urllib3\connectionpool.py", line 789, in urlopen response = self._make_request( File "C:\Users\20210595\.conda\envs\tableshift\lib\site-packages\urllib3\connectionpool.py", line 495, in _make_request conn.request( File "C:\Users\20210595\.conda\envs\tableshift\lib\site-packages\urllib3\connection.py", line 455, in request self.send(chunk) File "C:\Users\20210595\.conda\envs\tableshift\lib\http\client.py", line 972, in send self.sock.sendall(data) File "C:\Users\20210595\.conda\envs\tableshift\lib\ssl.py", line 1237, in sendall v = self.send(byte_view[count:]) File "C:\Users\20210595\.conda\envs\tableshift\lib\ssl.py", line 1206, in send return self._sslobj.write(data) OverflowError: string longer than 2147483647 bytes
What is the constraint for the size of datasets in OpenML? I could not find it (maybe I did not look long enough)
Is there a way to avoid such limitations?
Thanks
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with publish_dataset.py and the openml/base.py publish entry point, then trace the request through openml/_api_calls.py to the requests POST shown in the traceback. Reproduce with the reported 2.7GB dataset and determine the applicable dataset-size constraint or supported way to avoid the failure; document the confirmed limit and resulting guidance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100