ClickHouse / ClickHouse/sql.clickhouse.com

pypi.projects table seems to be missing data

Offen
#32 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Shell
Sterne
21
Forks
7
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

You've got a really great project here that we've found very helpful in tracking our library usage but we've recently encountered a bit of a bug.

The latest version of at least the two packages that I've checked seems to be incorrect. This seems to be caused by missing data in your pypi.projects table. For umap-learn it seems to have stopped updating in 2022-04. We've had 8 releases between then and now. It's not a general database problem since I've checked our evoc package and it has continual updates from it's creation in 2024 through 2026. It's also not an isolated case in the at our hdbscan project last had been updated in 2022-10 (we've had 13 releases between then and now).

A few links and bits of code that might help reproduce these findings are:

https://clickpy.clickhouse.com/dashboard/umap-learn
claims that the latest version is 5.3.0
the Top Version analytic shows a 5.11 version being quite commonly in use.

https://pypi.org/project/umap-learn/#history
Shows that the latest version is 5.12 (but it's very recent so 5.11 is a very understandable latest observed version)

```
SELECT upload_time, name, version
FROM pypi.projects
WHERE name = ('umap-learn')
order by upload_time descending
limit 2

```
```
| upload_time | name | version |
| --- | --- | --- |
| 2022-04-13 21:20:36.488 | umap-learn | 0.5.3 |
| 2021-02-08 21:04:43.727 | umap-learn | 0.5.1 |
```

hdbscan results
```
| upload_time | name | version |
| --- | --- | --- |
| 2022-10-31 17:23:44.432 | hdbscan | 0.8.29 |
| 2022-02-08 15:45:27.037 | hdbscan | 0.8.28 |
```

evoc results
```
| upload_time | name | version |
| --- | --- | --- |
| 2026-03-27 19:12:49.315 | evoc | 0.3.1 |
| 2026-03-27 19:12:48.050 | evoc | 0.3.1 |
```

I looked into it a bit and it looks like your pypi.pypi_downloads_per_day_by_version is being kept up to date resulting in our more recent versions still showing up in your Top Versions query.
```
SELECT
version AS name,
sum(count) AS value
FROM pypi.pypi_downloads_per_day_by_version
WHERE (date >= {min_date:String}::Date32) AND (date < {max_date:String}::Date32) AND (project = 'umap-learn')
AND 1=1 AND 1=1 AND 1=1
GROUP BY version ORDER BY value DESC LIMIT 6
```
```
| name | value |
| --- | --- |
| 0.5.7 | 19848394 |
| 0.5.3 | 14609667 |
| 0.5.9.post2 | 10838075 |
| 0.5.6 | 9860975 |
| 0.5.11 | 7156607 |
| 0.5.2 | 6382952 |
```

Hopefully this help track down the bug in the database because your work is really valuable to the open source community.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Beginne damit, die Abweichung in pypi.projects für umap-learn und hdbscan zu reproduzieren, und vergleiche sie dann mit pypi.pypi_downloads_per_day_by_version und den zugehörigen Abfragen, die im Bericht gezeigt werden. Verfolge den Ingestion-Pfad, der pypi.projects befüllt, und ermittle, warum die Aktualisierungen für diese Projekte beendet werden. Als erledigt gilt die Aufgabe, wenn aktuelle PyPI-Releases in pypi.projects erscheinen und die vorhandenen Download-Daten konsistent bleiben.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
clickhouse, sql
Bereich
data, databases
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Ruhig
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
45/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.