LZMA filtering documentation very poor
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 77.2k
- Forks
- 35.9k
- PR merge metrics
- PR metrics pending
Description
I'm trying to optimise compressing some documents, so I'm using LZMA's filters. These are really badly documented: https://docs.python.org/3/library/lzma.html#filter-chain-specs
General problems:
- The filters are using short-hand for variable names
lc,lp,mf, which is really bad practice for a publically available interface (and even for internal ones!). (ok, so this isn't a doc problem, but a library one). - It's not at all clear what any of the variables do. There's no links to any documentation (even third-party) which explain what any of these things do.
- It's not clear what values are acceptable for some of them.
-
presetis documented elsewhere, and I assume the value range is the same?
-
lcdoesn't say the maximum is 4, you have to readlpto see the combined maximum is 4, and then deduce.
-
lc,lp,pc,nice_len, don't state minimums
- No defaults stated other than
depth (0), - It's not at all clear what the different Filters do.
- The limited blurb that does exist about Filter chains seems to assume a high degree of expertise in these things. For example:
The last filter in the chain must be a compression filter, and any other filters must be delta or BCJ filters.- that has cleared nothing up. - What's a
MODE_FASTorMODE_NORMALdo? - What's the difference between
MF_HC3, MF_HC4, MF_BT2, MF_BT3, or MF_BT4? - What does
lzma.PRESET_EXTREMEdo? There's no documentation forlzma.PRESET_DEFAULTat all (which my IDE exposed to me via autocomplete).
Etc etc.
Some specific problems as well:
-
'filters': [{'id': 33, 'preset': 6, 'dict_size': 262144, 'lc': 1, 'lp': 1, 'mode': 2, 'nice_len': 1, 'mf': 3}]
That creates anLZMAError. It's only after generating thousands of combinations of these that I notice a pattern:nice_len: 1. It seems there's a minimum nice_len size. This is not documented. -
'filters': [{'id': 33, 'preset': 2147483653, 'dict_size': 1073741824, 'lc': 1, 'lp': 0, 'mode': 2, 'nice_len': 201, 'mf': 20}]
Generates aMemoryError(as do lots of others of my tests). No idea why as I have plenty of RAM. I presume some combination of those settings are causing an issue, but there's no documentation warning about this.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the LZMA filter-chain specifications section at the linked Python documentation page and review the descriptions of filter options, presets, modes, and filter chains. Document the parameter meanings, valid ranges, defaults, filter differences, and relevant error or memory constraints so the examples and failure cases are explained.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100