lionheart / lionheart/git-bigstore

Bigstore-compress filter is broken with S3 backend

Open
#32 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

help wanted
Dominant language
Python
Stars
201
Forks
28
PR merge metrics
No merged PRs in 30d

Description

I have not tested the other backends.

The first issue is the use of tempfile.TemporaryFile instead of tempfile.NamedTemporaryFile. The temporary file does not have a name on disk on most operating systems, which results in an error:

$ git bigstore push
pulling bigstore metadata...done
compressed!
Traceback (most recent call last):
  File "/home/jansel/testrepo/venv/bin/git-bigstore", line 87, in <module>
    args = parser.parse_args()
  File "/usr/lib/python2.7/argparse.py", line 1701, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/usr/lib/python2.7/argparse.py", line 1733, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/lib/python2.7/argparse.py", line 1942, in _parse_known_args
    stop_index = consume_positionals(start_index)
  File "/usr/lib/python2.7/argparse.py", line 1898, in consume_positionals
    take_action(action, args)
  File "/usr/lib/python2.7/argparse.py", line 1807, in take_action
    action(self, namespace, argument_values, option_string)
  File "/usr/lib/python2.7/argparse.py", line 1096, in __call__
    subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
  File "/usr/lib/python2.7/argparse.py", line 1733, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/lib/python2.7/argparse.py", line 1942, in _parse_known_args
    stop_index = consume_positionals(start_index)
  File "/usr/lib/python2.7/argparse.py", line 1898, in consume_positionals
    take_action(action, args)
  File "/usr/lib/python2.7/argparse.py", line 1807, in take_action
    action(self, namespace, argument_values, option_string)
  File "/home/jansel/testrepo/venv/bin/git-bigstore", line 33, in __call__
    push()
  File "/home/jansel/testrepo/venv/local/lib/python2.7/site-packages/bigstore/bigstore.py", line 246, in push
    backend.push(compressed_file, hexdigest, cb=ProgressPercentage(filename))
  File "/home/jansel/testrepo/venv/local/lib/python2.7/site-packages/bigstore/backends/s3.py", line 35, in push
    self.s3_client.upload_file(file.name, self.bucket, self.get_remote_file_name(hash), Callback=cb)
  File "/home/jansel/testrepo/venv/local/lib/python2.7/site-packages/boto3/s3/inject.py", line 106, in upload_file
    extra_args=ExtraArgs, callback=Callback)
  File "/home/jansel/testrepo/venv/local/lib/python2.7/site-packages/boto3/s3/transfer.py", line 275, in upload_file
    future.result()
  File "/home/jansel/testrepo/venv/local/lib/python2.7/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/home/jansel/testrepo/venv/local/lib/python2.7/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
OSError: [Errno 2] No such file or directory: '<fdopen>'

I fixed this issue on my fork, but bigstore-compress still seems buggy even after that. I observed the following behavior (with my partially fixed version):

  1. echo '*.bin filter=bigstore-compress' >.gitattributes
  2. dd if=/dev/zero of=test.bin bs=1M count=1
  3. git a . && git commit -m 'test' && git bigstore push runs without error
  4. git pull && git bigstore pull on another checkout results in a zero byte file instead of the expected 1M file

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with bigstore/bigstore.py and bigstore/backends/s3.py, following the traceback from git bigstore push and the S3 upload. Reproduce the .gitattributes, test.bin, push, pull, and checkout sequence described in the issue. Done means S3 push no longer fails on the temporary file and the pulled file retains its original 1M size.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, git, python
Domain
backend, cli, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.