hpc / hpc/mpifileutils

Feature request : dtar writes multiple files like tar | split pipeline

Open
#504 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
200
Forks
85
Avg merge
3d 21h
Merged PRs (30d)
2

Description

Hello mpifileutils developpers,

When tar'ing very big directories, tar files may exceed system accepted size for a file (typically 500 GB).
This is why one can tar and pipe with split such as
tar --verbose --create dir | split -d -b 512000m - fictar

I tried dtar, which is remarkable already on my PC, but I failed to do the same thing.
I could only write one file
mpirun -np 8 dtar --create --file fictar.tar dir
BTW the --verbose option had to be removed which contradicts the man.

Do you think my request is reasonable ? Does it make sense ?

Thanks a lot.

Olivier Cessenat

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or test is named. Start by reviewing dtar's --create and --file handling and the documented --verbose option, then compare the requested behavior with the tar | split -d -b pipeline. Define the output-file interface and size semantics before implementation; done means dtar can produce and document multiple files for one archive.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
cli, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.