scikit-learn / scikit-learn/scikit-learn
BUG: Test failure on ppc64le
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 67.3k
- Forks
- 27.4k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 58
Description
Describe the bug
While working on https://github.com/conda-forge/scikit-learn-feedstock/pull/123, I found that manifold/tests/test_t_sne.py::test_uniform_grid[barnes_hut] fails on ppc64le; @rth asked me to open a bug here.
=========================== short test summary info ============================
FAILED manifold/tests/test_t_sne.py::test_uniform_grid[barnes_hut] - Assertio...
= 1 failed, 13718 passed, 398 skipped, 6 xfailed, 1674 warnings in 382.15s (0:06:22) =
Here are the logs, the failure is the same across cpython 3.6-3.8:
Interestingly, PyPy doesn't have that failure (but has some segfaults instead..., see #16956).
A more detailed trace is below the fold. From what I can see from the log, the failure seems to be a question of "only" the quality of an embedding, and not a hard failure per se.
=================================== FAILURES ===================================
________________________ test_uniform_grid[barnes_hut] _________________________
[gw2] linux -- Python 3.6.10 $PREFIX/bin/python
method = 'barnes_hut'
@pytest.mark.parametrize('method', ['barnes_hut', 'exact'])
def test_uniform_grid(method):
"""Make sure that TSNE can approximately recover a uniform 2D grid
Due to ties in distances between point in X_2d_grid, this test is platform
dependent for ``method='barnes_hut'`` due to numerical imprecision.
Also, t-SNE is not assured to converge to the right solution because bad
initialization can lead to convergence to bad local minimum (the
optimization problem is non-convex). To avoid breaking the test too often,
we re-run t-SNE from the final point when the convergence is not good
enough.
"""
seeds = [0, 1, 2]
n_iter = 500
for seed in seeds:
tsne = TSNE(n_components=2, init='random', random_state=seed,
perplexity=20, n_iter=n_iter, method=method)
Y = tsne.fit_transform(X_2d_grid)
try_name = "{}_{}".format(method, seed)
try:
> assert_uniform_grid(Y, try_name)
../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.6/site-packages/sklearn/manifold/tests/test_t_sne.py:784:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Y = array([[ 52.326397 , -15.92225 ],
[ 46.679527 , -20.175953 ],
[ 40.870537 , -24.181147 ],
...[-35.291374 , 22.122814 ],
[-42.2738 , 18.793724 ],
[-48.922283 , 15.606232 ]], dtype=float32)
try_name = 'barnes_hut_1'
def assert_uniform_grid(Y, try_name=None):
# Ensure that the resulting embedding leads to approximately
# uniformly spaced points: the distance to the closest neighbors
# should be non-zero and approximately constant.
nn = NearestNeighbors(n_neighbors=1).fit(Y)
dist_to_nn = nn.kneighbors(return_distance=True)[0].ravel()
assert dist_to_nn.min() > 0.1
smallest_to_mean = dist_to_nn.min() / np.mean(dist_to_nn)
largest_to_mean = dist_to_nn.max() / np.mean(dist_to_nn)
assert smallest_to_mean > .5, try_name
> assert largest_to_mean < 2, try_name
E AssertionError: barnes_hut_1
E assert 6.67359409617653 < 2
../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.6/site-packages/sklearn/manifold/tests/test_t_sne.py:807: AssertionError
During handling of the above exception, another exception occurred:
method = 'barnes_hut'
@pytest.mark.parametrize('method', ['barnes_hut', 'exact'])
def test_uniform_grid(method):
"""Make sure that TSNE can approximately recover a uniform 2D grid
Due to ties in distances between point in X_2d_grid, this test is platform
dependent for ``method='barnes_hut'`` due to numerical imprecision.
Also, t-SNE is not assured to converge to the right solution because bad
initialization can lead to convergence to bad local minimum (the
optimization problem is non-convex). To avoid breaking the test too often,
we re-run t-SNE from the final point when the convergence is not good
enough.
"""
seeds = [0, 1, 2]
n_iter = 500
for seed in seeds:
tsne = TSNE(n_components=2, init='random', random_state=seed,
perplexity=20, n_iter=n_iter, method=method)
Y = tsne.fit_transform(X_2d_grid)
try_name = "{}_{}".format(method, seed)
try:
assert_uniform_grid(Y, try_name)
except AssertionError:
# If the test fails a first time, re-run with init=Y to see if
# this was caused by a bad initialization. Note that this will
# also run an early_exaggeration step.
try_name += ":rerun"
tsne.init = Y
Y = tsne.fit_transform(X_2d_grid)
> assert_uniform_grid(Y, try_name)
../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.6/site-packages/sklearn/manifold/tests/test_t_sne.py:792:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Y = array([[-18.169476 , 6.0802336 ],
[-18.278513 , 2.8822129 ],
[-18.671782 , -0.4646889 ],
...[ 22.550077 , 19.698557 ],
[ 21.399723 , 22.933178 ],
[ 16.22136 , 28.22955 ]], dtype=float32)
try_name = 'barnes_hut_1:rerun'
def assert_uniform_grid(Y, try_name=None):
# Ensure that the resulting embedding leads to approximately
# uniformly spaced points: the distance to the closest neighbors
# should be non-zero and approximately constant.
nn = NearestNeighbors(n_neighbors=1).fit(Y)
dist_to_nn = nn.kneighbors(return_distance=True)[0].ravel()
assert dist_to_nn.min() > 0.1
smallest_to_mean = dist_to_nn.min() / np.mean(dist_to_nn)
largest_to_mean = dist_to_nn.max() / np.mean(dist_to_nn)
assert smallest_to_mean > .5, try_name
> assert largest_to_mean < 2, try_name
E AssertionError: barnes_hut_1:rerun
E assert 2.145051767903112 < 2
../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.6/site-packages/sklearn/manifold/tests/test_t_sne.py:807: AssertionError
=============================== warnings summary ===============================
[...]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with manifold/tests/test_t_sne.py::test_uniform_grid[barnes_hut] and reproduce the failure on ppc64le across the reported CPython versions. Compare the Barnes-Hut results and test behavior with the linked logs and the PyPy result. Done means the ppc64le test failure is resolved while the uniform-grid assertion continues to validate the embedding.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100