pulp / pulp/pulpcore

Pulp export chunks are seen by suspicious by virus scanners because tar output is piped into split

Open
#3,236 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Issue Katello satellite
Dominant language
Python
Stars
598
Forks
168
Avg merge
1d 4h
Merged PRs (30d)
86

Description

Version
All

Describe the bug
See attached BZ for full details.

The gzipped tarball is piped into the split command to create the chunks. Every chunk ( or split ) after the first one does not have file header data to indicate the type of file, so the virus scanner sees it as random data.

When creating a chunked tar archive on my local machine and piping it through "split", only the first chunk is recognised with the file command:

woody@woody-arch-thinkpad ~/Documents $ tar -cvzf - file1.mp4 file2.mp4 | split -b 2MB - test.tar.gz.
woody@woody-arch-thinkpad ~/Documents $ file test.tar.gz.aa 
test.tar.gz.aa: gzip compressed data, from Unix, original size modulo 2^32 842674519
woody@woody-arch-thinkpad ~/Documents $ file test.tar.gz.ab
test.tar.gz.ab: data

If I use the "--tape-length" option, I can use the "file" command to identify each chunk as a tarball. I assume the virus scanning tool would be using "file" or similar methods.

woody@woody-arch-thinkpad ~/Documents $ tar -cv --tape-length=2000 --file=test-archive-{0..100}.tar file1.mp4 file2.mp4
woody@woody-arch-thinkpad ~/Documents $ file test-archive-0.tar 
test-archive-0.tar: POSIX tar archive (GNU)
woody@woody-arch-thinkpad ~/Documents $ file test-archive-1.tar 
test-archive-1.tar: tar archive

A drawback is that those "--tape-length" archives can't be gzipped from that single tar command. It gives an error
"tar: Cannot use multi-volume compressed archives"

Can we request that the exports are chunked with "--tape-length" instead of piping into the "split" command?

Additional context
https://bugzilla.redhat.com/show_bug.cgi?id=2109742

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or test is named in the issue. Start by locating the export implementation that pipes tar output through split, then reproduce the chunking behavior described with file and a virus scanner; done means exported chunks are individually recognized without breaking archive reconstruction.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.