lionheart / lionheart/git-bigstore
Bigstore-compress filter is broken with S3 backend
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 201
- Forks
- 28
- PR merge metrics
- No merged PRs in 30d
Description
I have not tested the other backends.
The first issue is the use of tempfile.TemporaryFile instead of tempfile.NamedTemporaryFile. The temporary file does not have a name on disk on most operating systems, which results in an error:
$ git bigstore push
pulling bigstore metadata...done
compressed!
Traceback (most recent call last):
File "/home/jansel/testrepo/venv/bin/git-bigstore", line 87, in <module>
args = parser.parse_args()
File "/usr/lib/python2.7/argparse.py", line 1701, in parse_args
args, argv = self.parse_known_args(args, namespace)
File "/usr/lib/python2.7/argparse.py", line 1733, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
File "/usr/lib/python2.7/argparse.py", line 1942, in _parse_known_args
stop_index = consume_positionals(start_index)
File "/usr/lib/python2.7/argparse.py", line 1898, in consume_positionals
take_action(action, args)
File "/usr/lib/python2.7/argparse.py", line 1807, in take_action
action(self, namespace, argument_values, option_string)
File "/usr/lib/python2.7/argparse.py", line 1096, in __call__
subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
File "/usr/lib/python2.7/argparse.py", line 1733, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
File "/usr/lib/python2.7/argparse.py", line 1942, in _parse_known_args
stop_index = consume_positionals(start_index)
File "/usr/lib/python2.7/argparse.py", line 1898, in consume_positionals
take_action(action, args)
File "/usr/lib/python2.7/argparse.py", line 1807, in take_action
action(self, namespace, argument_values, option_string)
File "/home/jansel/testrepo/venv/bin/git-bigstore", line 33, in __call__
push()
File "/home/jansel/testrepo/venv/local/lib/python2.7/site-packages/bigstore/bigstore.py", line 246, in push
backend.push(compressed_file, hexdigest, cb=ProgressPercentage(filename))
File "/home/jansel/testrepo/venv/local/lib/python2.7/site-packages/bigstore/backends/s3.py", line 35, in push
self.s3_client.upload_file(file.name, self.bucket, self.get_remote_file_name(hash), Callback=cb)
File "/home/jansel/testrepo/venv/local/lib/python2.7/site-packages/boto3/s3/inject.py", line 106, in upload_file
extra_args=ExtraArgs, callback=Callback)
File "/home/jansel/testrepo/venv/local/lib/python2.7/site-packages/boto3/s3/transfer.py", line 275, in upload_file
future.result()
File "/home/jansel/testrepo/venv/local/lib/python2.7/site-packages/s3transfer/futures.py", line 73, in result
return self._coordinator.result()
File "/home/jansel/testrepo/venv/local/lib/python2.7/site-packages/s3transfer/futures.py", line 233, in result
raise self._exception
OSError: [Errno 2] No such file or directory: '<fdopen>'
I fixed this issue on my fork, but bigstore-compress still seems buggy even after that. I observed the following behavior (with my partially fixed version):
echo '*.bin filter=bigstore-compress' >.gitattributesdd if=/dev/zero of=test.bin bs=1M count=1git a . && git commit -m 'test' && git bigstore pushruns without errorgit pull && git bigstore pullon another checkout results in a zero byte file instead of the expected 1M file
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with bigstore/bigstore.py and bigstore/backends/s3.py, following the traceback from git bigstore push and the S3 upload. Reproduce the .gitattributes, test.bin, push, pull, and checkout sequence described in the issue. Done means S3 push no longer fails on the temporary file and the pulled file retains its original 1M size.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, git, python
- Domain
- backend, cli, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100