tslearn-team / tslearn-team/tslearn

Inconsistent calculation results between tslearn, pyts and fastdtw

Open
#274 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
3.2k
Forks
384
Avg merge
3d 12h
Merged PRs (30d)
11

Description

Description

Please forgive me for my poor English since English is not my native language.

I have read the source code and Sakoe_Chiba band generation examples from tslearn, pyts. The generation manner of the Sakoe_Chiba seems different between pyts and tslearn, which leads to different calculation results when comparing 2 sequences with different lengths. I have also read the source code of fastdtw(https://pypi.org/project/fastdtw/) when the radius parameter is different, the calculation results between fastdtw and pyts are also varied.

Codes
import numpy as np
from tslearn.metrics import dtw as ts_dtw
from pyts.metrics import dtw as py_dtw
from fastdtw import fastdtw

if __name == "__main__":
    np.random.seed(2020)
    seq_0 = np.random.randn(140)
    seq_1 = np.random.randn(50)

    # Experiment 1: Different DTW caculation(Consistent)
    # [INFO] DTW Calculation:
    # -- tslearn dtw: 8.26240
    # -- pyts dtw: 8.26240
    print("[INFO] DTW Calculation:")
    print("-- tslearn dtw: {:.5f}".format(ts_dtw(seq_0, seq_1)))
    print("-- pyts dtw: {:.5f}".format(py_dtw(seq_0, seq_1)))
py_dtw
    # Experiment 2: FastDTW calculation(Inconsistent)
    # [INFO] FastDTW Calculation:
    # -- FastDTW results: 8.67608
    # -- pyts FastDTW: 9.12243
    print("\n[INFO] FastDTW Calculation:")
    print("-- FastDTW results: {:.5f}".format(
        np.sqrt(fastdtw(seq_0, seq_1, radius=2, dist=lambda x, y: (x-y)**2)[0])))
    print("-- pyts FastDTW: {:.5f}".format(
        py_dtw(seq_0, seq_1, method="fast", options={"radius": 2})))

    # Experiment 3: Sakoe_Chiba calculation(Inconsistent)
    # [INFO] Sakoe_Chiba Calculation:
    # -- tslearn Sakoe_Chiba dtw: 8.26240
    # -- pyts Sakoe_Chiba dtw: 10.49161
    print("\n[INFO] Sakoe_Chiba Calculation:")
    print("-- tslearn Sakoe_Chiba dtw: {:.5f}".format(
        ts_dtw(seq_0, seq_1, sakoe_chiba_radius=5)))
    print("-- pyts Sakoe_Chiba dtw: {:.5f}".format(
        py_dtw(seq_0, seq_1, method="sakoechiba",  options={"window_size": 5})))

    # Experiment 4: itakura calculation(In this example, they are consistent, however, I haven't read the source code yet)
    # [INFO] itakura Calculation:
    # -- tslearn itakura dtw: 8.51087
    # -- pyts itakura dtw: 8.51087
    print("\n[INFO] itakura Calculation:")
    print("-- tslearn itakura dtw: {:.5f}".format(
        ts_dtw(seq_0, seq_1, itakura_max_slope=6)))
    print("-- pyts itakura dtw: {:.5f}".format(
        py_dtw(seq_0, seq_1, method="itakura",  options={"max_slope": 6})))

Versions

OS: Ubuntu 18.04.4 LTS
NumPy 1.18.1
SciPy 1.4.1
Scikit-Learn 0.22.1
Numba 0.49.1
Pyts 0.11.0
tslearn: '0.4.1'
fastdtw: See pypi

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the supplied Python example and comparing the tslearn.metrics.dtw and pyts.metrics.dtw calls for FastDTW and Sakoe-Chiba constraints. Then inspect the relevant DTW entry points and constraint handling; done requires determining the intended behavior and resolving or documenting the discrepancies.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.