scikit-learn / scikit-learn/scikit-learn

BUG: Test failure on ppc64le

Open
#16,955 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Bug help wanted module:test-suite
Dominant language
Python
Stars
67.3k
Forks
27.4k
Avg merge
1d 15h
Merged PRs (30d)
58

Description

Describe the bug

While working on https://github.com/conda-forge/scikit-learn-feedstock/pull/123, I found that manifold/tests/test_t_sne.py::test_uniform_grid[barnes_hut] fails on ppc64le; @rth asked me to open a bug here.

=========================== short test summary info ============================
FAILED manifold/tests/test_t_sne.py::test_uniform_grid[barnes_hut] - Assertio...
= 1 failed, 13718 passed, 398 skipped, 6 xfailed, 1674 warnings in 382.15s (0:06:22) =

Here are the logs, the failure is the same across cpython 3.6-3.8:

Interestingly, PyPy doesn't have that failure (but has some segfaults instead..., see #16956).

A more detailed trace is below the fold. From what I can see from the log, the failure seems to be a question of "only" the quality of an embedding, and not a hard failure per se.

=================================== FAILURES ===================================
________________________ test_uniform_grid[barnes_hut] _________________________
[gw2] linux -- Python 3.6.10 $PREFIX/bin/python

method = 'barnes_hut'

    @pytest.mark.parametrize('method', ['barnes_hut', 'exact'])
    def test_uniform_grid(method):
        """Make sure that TSNE can approximately recover a uniform 2D grid
    
        Due to ties in distances between point in X_2d_grid, this test is platform
        dependent for ``method='barnes_hut'`` due to numerical imprecision.
    
        Also, t-SNE is not assured to converge to the right solution because bad
        initialization can lead to convergence to bad local minimum (the
        optimization problem is non-convex). To avoid breaking the test too often,
        we re-run t-SNE from the final point when the convergence is not good
        enough.
        """
        seeds = [0, 1, 2]
        n_iter = 500
        for seed in seeds:
            tsne = TSNE(n_components=2, init='random', random_state=seed,
                        perplexity=20, n_iter=n_iter, method=method)
            Y = tsne.fit_transform(X_2d_grid)
    
            try_name = "{}_{}".format(method, seed)
            try:
>               assert_uniform_grid(Y, try_name)

../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.6/site-packages/sklearn/manifold/tests/test_t_sne.py:784: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

Y = array([[ 52.326397  , -15.92225   ],
       [ 46.679527  , -20.175953  ],
       [ 40.870537  , -24.181147  ],
       ...[-35.291374  ,  22.122814  ],
       [-42.2738    ,  18.793724  ],
       [-48.922283  ,  15.606232  ]], dtype=float32)
try_name = 'barnes_hut_1'

    def assert_uniform_grid(Y, try_name=None):
        # Ensure that the resulting embedding leads to approximately
        # uniformly spaced points: the distance to the closest neighbors
        # should be non-zero and approximately constant.
        nn = NearestNeighbors(n_neighbors=1).fit(Y)
        dist_to_nn = nn.kneighbors(return_distance=True)[0].ravel()
        assert dist_to_nn.min() > 0.1
    
        smallest_to_mean = dist_to_nn.min() / np.mean(dist_to_nn)
        largest_to_mean = dist_to_nn.max() / np.mean(dist_to_nn)
    
        assert smallest_to_mean > .5, try_name
>       assert largest_to_mean < 2, try_name
E       AssertionError: barnes_hut_1
E       assert 6.67359409617653 < 2

../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.6/site-packages/sklearn/manifold/tests/test_t_sne.py:807: AssertionError

During handling of the above exception, another exception occurred:

method = 'barnes_hut'

    @pytest.mark.parametrize('method', ['barnes_hut', 'exact'])
    def test_uniform_grid(method):
        """Make sure that TSNE can approximately recover a uniform 2D grid
    
        Due to ties in distances between point in X_2d_grid, this test is platform
        dependent for ``method='barnes_hut'`` due to numerical imprecision.
    
        Also, t-SNE is not assured to converge to the right solution because bad
        initialization can lead to convergence to bad local minimum (the
        optimization problem is non-convex). To avoid breaking the test too often,
        we re-run t-SNE from the final point when the convergence is not good
        enough.
        """
        seeds = [0, 1, 2]
        n_iter = 500
        for seed in seeds:
            tsne = TSNE(n_components=2, init='random', random_state=seed,
                        perplexity=20, n_iter=n_iter, method=method)
            Y = tsne.fit_transform(X_2d_grid)
    
            try_name = "{}_{}".format(method, seed)
            try:
                assert_uniform_grid(Y, try_name)
            except AssertionError:
                # If the test fails a first time, re-run with init=Y to see if
                # this was caused by a bad initialization. Note that this will
                # also run an early_exaggeration step.
                try_name += ":rerun"
                tsne.init = Y
                Y = tsne.fit_transform(X_2d_grid)
>               assert_uniform_grid(Y, try_name)

../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.6/site-packages/sklearn/manifold/tests/test_t_sne.py:792: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

Y = array([[-18.169476  ,   6.0802336 ],
       [-18.278513  ,   2.8822129 ],
       [-18.671782  ,  -0.4646889 ],
       ...[ 22.550077  ,  19.698557  ],
       [ 21.399723  ,  22.933178  ],
       [ 16.22136   ,  28.22955   ]], dtype=float32)
try_name = 'barnes_hut_1:rerun'

    def assert_uniform_grid(Y, try_name=None):
        # Ensure that the resulting embedding leads to approximately
        # uniformly spaced points: the distance to the closest neighbors
        # should be non-zero and approximately constant.
        nn = NearestNeighbors(n_neighbors=1).fit(Y)
        dist_to_nn = nn.kneighbors(return_distance=True)[0].ravel()
        assert dist_to_nn.min() > 0.1
    
        smallest_to_mean = dist_to_nn.min() / np.mean(dist_to_nn)
        largest_to_mean = dist_to_nn.max() / np.mean(dist_to_nn)
    
        assert smallest_to_mean > .5, try_name
>       assert largest_to_mean < 2, try_name
E       AssertionError: barnes_hut_1:rerun
E       assert 2.145051767903112 < 2

../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.6/site-packages/sklearn/manifold/tests/test_t_sne.py:807: AssertionError
=============================== warnings summary ===============================
[...]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with manifold/tests/test_t_sne.py::test_uniform_grid[barnes_hut] and reproduce the failure on ppc64le across the reported CPython versions. Compare the Barnes-Hut results and test behavior with the linked logs and the PyPy result. Done means the ppc64le test failure is resolved while the uniform-grid assertion continues to validate the embedding.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.