Lightning-AI / Lightning-AI/torchmetrics

Add a CW-SSIM support for torchmetrics

Open
#2,428 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement New metric
Dominant language
Python
Stars
2.5k
Forks
526
Avg merge
6d 11h
Merged PRs (30d)
5

Description

## 🚀 Feature

A Complex-Wavelets Structure Similarity (also know as CW-SSIM) support with Steerable Pyramid (SP) or Dual-Tree Complex Wavelet Transform Method (DTCWT). Maybe support all possible Q-shift and first level filters as well?

### Motivation

I noticed that someone just mentioned in #799
> Would you accept a PR for one not on the list? I have an implementation of complex wavelet structural similarity (CW-SSIM) that I could contribute. (https://ieeexplore.ieee.org/abstract/document/5109651)

For some research purpose, I found that there is a few project with CW-SSIM code but they are not updated to the latest pytorch version. Will torchmetrics add a CW-SSIM support in `torchmetrics.image`? Here are some collections with old codes.

* https://github.com/jterrace/pyssim
* https://github.com/LeonArcher/py_cwt2d
* https://github.com/dingkeyan93/IQA-optimization
* https://github.com/fbcotter/pytorch_wavelets
* https://github.com/LabForComputationalVision/pyrtools
* https://github.com/LabForComputationalVision/plenoptic

I just tried to use latest pytorch version (which has supported complex convolution and it's important to my usage) to achieve this function. It's very difficult for me to understand the math formula for all the complex wavelet things. If there are some suggestions on math I would appreciate for that. Please do not make confuse with Complex Wavelet Transform and Continuous Wavelet Transform because of the `CWT` abbreviation.

### Pitch

* CW-SSIM module and functional support.
* Steerable Pyramid (SP) or Dual-Tree Complex Wavelet Transform (DTCWT) Method (Better for both, A thesis about their performance: https://link.springer.com/chapter/10.1007/978-3-642-13681-8_42)
* Q-shift filters and other first level filters if use DTCWT method according to matlab's docs. (https://www.mathworks.com/help/wavelet/ref/dualtree2.html)

### Alternatives

I tried the scipy and pywavelets but they do not support SP or DTCWT. Only Discrete Wavelet Transform (DWT) with multi-dimensional support and Continuous Wavelet Transform are included. Listed projects are too old to run on latest PyTorch.

### Additional context

If there are some further math discussion about SP or DTCWT, I'll try to achieve myself and pull request to the torchmetrics. I'm very confused with the relationship of scaling function and wavelet function and whether it should be considered in SP or DTCWT. How does the SP or DTCWT's filters be calucated? Sorry for my pool math about the discrete and continous domain conversion.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the torchmetrics.image area and review the CW-SSIM references and implementations linked in the issue. Clarify whether support should use a Steerable Pyramid or DTCWT, including filter options, before defining the functional and module APIs. Done means documented, tested CW-SSIM support for the selected method.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.