dask / dask/dask

Mamba and conda solve of CI environment files installs ancient s3fs version causing poor performance

Open
#10,276 9 comments 1 reaction 0 assignees View on GitHub
needs attention needs triage
Dominant language
Python
Stars
13.9k
Forks
2k
PR merge metrics
No merged PRs in 30d

Description

When installing our test-environments that we are using in CI I am pulling in an ancient version of `s3fs` that is causing severe performance problems, see https://github.com/dask/dask/issues/10275 and https://github.com/fsspec/filesystem_spec/issues/1261

It should not be that easy to pull in an ancient version of such a critical library. Is there any way we can set lower pinnings for this? I don't know what's the cause of this pinning so I don't know how common this actually is when users install their environment

I haven't checked yet what version is installed in our CI jobs

conda export

```
# This file may be used to create an environment using:
# $ conda create --name --file
# platform: osx-arm64
aiohttp=3.8.4=py310h8e9501a_0
aiosignal=1.3.1=pyhd8ed1ab_0
aom=3.5.0=h7ea286d_0
appnope=0.1.3=pyhd8ed1ab_0
arrow-cpp=10.0.1=hce30654_23_cpu
asciitree=0.3.3=py_2
asttokens=2.2.1=pyhd8ed1ab_0
async-timeout=4.0.2=pyhd8ed1ab_0
atk-1.0=2.38.0=hcb7b3dd_1
attrs=23.1.0=pyh71513ae_1
aws-c-auth=0.6.26=h0c2019c_6
aws-c-cal=0.5.26=h9c01cc2_0
aws-c-common=0.8.17=hb547adb_0
aws-c-compression=0.2.16=h7a7fafa_6
aws-c-event-stream=0.2.20=heb79caa_6
aws-c-http=0.7.7=h5140e40_3
aws-c-io=0.13.21=hc2875bd_3
aws-c-mqtt=0.8.6=h34492a6_15
aws-c-s3=0.2.8=h28047f4_4
aws-c-sdkutils=0.1.9=h7a7fafa_1
aws-checksums=0.1.14=h7a7fafa_6
aws-crt-cpp=0.19.9=h1ca6c90_5
aws-sam-translator=1.66.0=pyhd8ed1ab_0
aws-sdk-cpp=1.10.57=hc8901a9_10
aws-xray-sdk=2.12.0=pyhd8ed1ab_0
backcall=0.2.0=pyh9f0ad1d_0
backports=1.0=pyhd8ed1ab_3
backports.functools_lru_cache=1.6.4=pyhd8ed1ab_0
bcrypt=3.2.2=py310h8e9501a_1
blinker=1.6.2=pyhd8ed1ab_0
blosc=1.21.3=h1d6ff8b_0
bokeh=3.1.1=pyhd8ed1ab_0
boto3=1.26.132=pyhd8ed1ab_0
botocore=1.29.132=pyhd8ed1ab_0
brotli=1.0.9=h1a8c8d9_8
brotli-bin=1.0.9=h1a8c8d9_8
brotlipy=0.7.0=py310h8e9501a_1005
brunsli=0.1=h9f76cd9_0
bzip2=1.0.8=h3422bc3_4
c-ares=1.18.1=h3422bc3_0
c-blosc2=2.8.0=h303ed30_1
ca-certificates=2023.5.7=hf0a4a13_0
cached-property=1.5.2=hd8ed1ab_1
cached_property=1.5.2=pyha770c72_1
cachey=0.2.1=pyh9f0ad1d_0
cairo=1.16.0=h4741ed9_1015
certifi=2023.5.7=pyhd8ed1ab_0
cffi=1.15.1=py310h2399d43_3
cfgv=3.3.1=pyhd8ed1ab_0
cfitsio=4.2.0=h2f961c4_0
cfn-lint=0.77.5=pyhd8ed1ab_0
charls=2.4.1=hb7217d7_0
charset-normalizer=2.1.1=pyhd8ed1ab_0
click=8.1.3=unix_pyhd8ed1ab_2
cloudpickle=2.2.1=pyhd8ed1ab_0
colorama=0.4.6=pyhd8ed1ab_0
comm=0.1.3=pyhd8ed1ab_0
contourpy=1.0.7=py310h2887b22_0
coverage=7.2.5=py310h2aa6e3c_0
cramjam=2.6.2=py310h1dc5ec9_0
crick=0.0.3=py310hf1a086a_1003
cryptography=40.0.2=py310hfc83b78_0
curl=8.0.1=heffe338_0
cytoolz=0.12.0=py310h8e9501a_1
dask-core=2023.4.1=pyhd8ed1ab_0
dav1d=1.0.0=he4db4b2_1
debugpy=1.6.7=py310h0f1eb42_0
decorator=5.1.1=pyhd8ed1ab_0
distlib=0.3.6=pyhd8ed1ab_0
distributed=2023.4.1+12.g2d80271d=pypi_0
docker-py=6.1.0=pyhd8ed1ab_0
ecdsa=0.18.0=pyhd8ed1ab_1
entrypoints=0.4=pyhd8ed1ab_0
exceptiongroup=1.1.1=pyhd8ed1ab_0
execnet=1.9.0=pyhd8ed1ab_0
executing=1.2.0=pyhd8ed1ab_0
expat=2.5.0=hb7217d7_1
fasteners=0.17.3=pyhd8ed1ab_0
fastparquet=2023.4.0=py310hf1a086a_0
filelock=3.12.0=pyhd8ed1ab_0
flask=2.3.2=pyhd8ed1ab_0
flask_cors=3.0.10=pyhd3deb0d_0
font-ttf-dejavu-sans-mono=2.37=hab24e00_0
font-ttf-inconsolata=3.000=h77eed37_0
font-ttf-source-code-pro=2.038=h77eed37_0
font-ttf-ubuntu=0.83=hab24e00_0
fontconfig=2.14.2=h82840c6_0
fonts-conda-ecosystem=1=0
fonts-conda-forge=1=0
freetype=2.12.1=hd633e50_1
fribidi=1.0.10=h27ca646_0
frozenlist=1.3.3=py310h8e9501a_0
fsspec=2023.5.0=pyh1a96a4e_0
gdk-pixbuf=2.42.10=h1ac0d0d_2
gettext=0.21.1=h0186832_0
gflags=2.2.2=hc88da5d_1004
giflib=5.2.1=h1a8c8d9_3
glog=0.6.0=h6da1cb0_0
gmp=6.2.1=h9f76cd9_0
gmpy2=2.1.2=py310h2e6cad2_1
graphite2=1.3.13=h9f76cd9_1001
graphql-core=3.2.3=pyhd8ed1ab_0
graphviz=8.0.5=h10878c0_0
greenlet=2.0.2=py310h0f1eb42_0
gtk2=2.24.33=h57013de_2
gts=0.7.6=h4b6d4d6_2
h5py=3.8.0=nompi_py310h3c889c5_101
harfbuzz=7.3.0=h46e5fef_0
hdf5=1.14.0=nompi_h6b85c65_103
heapdict=1.0.1=py_0
httpretty=1.1.4=pyhd8ed1ab_0
icu=72.1=he12128b_0
identify=2.5.24=pyhd8ed1ab_0
idna=3.4=pyhd8ed1ab_0
imagecodecs=2023.1.23=py310hcb660b6_2
imageio=2.27.0=pyh24c5eb1_0
importlib-metadata=6.6.0=pyha770c72_0
importlib_metadata=6.6.0=hd8ed1ab_0
importlib_resources=5.12.0=pyhd8ed1ab_0
iniconfig=2.0.0=pyhd8ed1ab_0
ipycytoscape=1.3.3=pyhd8ed1ab_1
ipykernel=6.21.2=pyh736e0ef_0
ipython=8.13.2=pyhd1c38e8_0
ipywidgets=8.0.4=pyhd8ed1ab_0
itsdangerous=2.1.2=pyhd8ed1ab_0
jedi=0.18.2=pyhd8ed1ab_0
jinja2=3.1.2=pyhd8ed1ab_1
jmespath=1.0.1=pyhd8ed1ab_0
joblib=1.2.0=pyhd8ed1ab_0
jschema-to-python=1.2.3=pyhd8ed1ab_0
jsondiff=2.0.0=pyhd8ed1ab_0
jsonpatch=1.32=pyhd8ed1ab_0
jsonpickle=2.2.0=pyhd8ed1ab_0
jsonpointer=2.0=py_0
jsonschema=4.17.3=pyhd8ed1ab_0
jsonschema-spec=0.1.4=pyhd8ed1ab_0
junit-xml=1.9=pyh9f0ad1d_0
jupyter_client=8.2.0=pyhd8ed1ab_0
jupyter_core=5.3.0=py310hbe9552e_0
jupyterlab_widgets=3.0.7=pyhd8ed1ab_0
jxrlib=1.1=h27ca646_2
krb5=1.20.1=h69eda48_0
lazy-object-proxy=1.9.0=py310h8e9501a_0
lazy_loader=0.2=pyhd8ed1ab_0
lcms2=2.15=hd835a16_1
lerc=4.0.0=h9a09cb3_0
libabseil=20230125.0=cxx17_hb7217d7_1
libaec=1.0.6=hb7217d7_1
libarrow=10.0.1=hb50b599_23_cpu
libavif=0.11.1=h3d80962_0
libblas=3.9.0=16_osxarm64_openblas
libbrotlicommon=1.0.9=h1a8c8d9_8
libbrotlidec=1.0.9=h1a8c8d9_8
libbrotlienc=1.0.9=h1a8c8d9_8
libcblas=3.9.0=16_osxarm64_openblas
libcrc32c=1.1.2=hbdafb3b_0
libcurl=8.0.1=heffe338_0
libcxx=16.0.3=h4653b0c_0
libdeflate=1.18=h1a8c8d9_0
libedit=3.1.20191231=hc8eb9b7_2
libev=4.33=h642e427_1
libevent=2.1.12=h5643135_0
libexpat=2.5.0=hb7217d7_1
libffi=3.4.2=h3422bc3_5
libgd=2.3.3=h939342b_6
libgfortran=5.0.0=12_2_0_hd922786_31
libgfortran5=12.2.0=h0eea778_31
libglib=2.76.2=h24e9cb9_0
libgoogle-cloud=2.10.0=he22f4c0_0
libgrpc=1.54.2=h9dbdbd0_0
libiconv=1.17=he4db4b2_0
libjpeg-turbo=2.1.5.1=h1a8c8d9_0
liblapack=3.9.0=16_osxarm64_openblas
libllvm11=11.1.0=hfa12f05_5
libnghttp2=1.52.0=hae82a92_0
libopenblas=0.3.21=openmp_hc731615_3
libpng=1.6.39=h76d750c_0
libprotobuf=3.21.12=hb5ab8b9_0
librsvg=2.56.0=h6c0e662_0
libsodium=1.0.18=h27ca646_1
libsqlite=3.41.2=hb31c410_1
libssh2=1.10.0=h7a5bd25_3
libthrift=0.18.1=ha061701_1
libtiff=4.5.0=h4f7d55c_6
libtool=2.4.7=hb7217d7_0
libutf8proc=2.8.0=h1a8c8d9_0
libwebp=1.3.0=h66d6964_0
libwebp-base=1.3.0=h1a8c8d9_0
libxcb=1.13=h9b22ae9_1004
libxml2=2.10.4=h2aff0a6_0
libzlib=1.2.13=h03a7124_4
libzopfli=1.0.3=h9f76cd9_0
llvm-openmp=16.0.3=h1c12783_0
llvmlite=0.39.1=py310h1e34944_1
locket=1.0.0=pyhd8ed1ab_0
lz4=4.3.2=py310ha6df754_0
lz4-c=1.9.4=hb7217d7_0
markupsafe=2.1.2=py310h8e9501a_0
matplotlib-inline=0.1.6=pyhd8ed1ab_0
mimesis=9.0.0=pyhd8ed1ab_0
mmh3=3.0.0=py310h0f1eb42_3
moto=4.1.9=pyhd8ed1ab_0
mpc=1.3.1=h91ba8db_0
mpfr=4.2.0=he09a6ba_0
mpmath=1.3.0=pyhd8ed1ab_0
msgpack-python=1.0.5=py310h2887b22_0
multidict=6.0.4=py310h8e9501a_0
nbformat=5.8.0=pyhd8ed1ab_0
ncurses=6.3=h07bb92c_1
nest-asyncio=1.5.6=pyhd8ed1ab_0
networkx=2.8.8=pyhd8ed1ab_0
nodeenv=1.7.0=pyhd8ed1ab_0
numba=0.56.4=py310h3124f1e_1
numcodecs=0.11.0=py310h0f1eb42_1
numexpr=2.8.4=py310h2b830bf_0
numpy=1.23.5=py310h5d7c261_0
openapi-schema-validator=0.4.4=pyhd8ed1ab_0
openapi-spec-validator=0.5.6=pyhd8ed1ab_0
openjpeg=2.5.0=hbc2ba62_2
openssl=3.1.0=h53f4e23_3
orc=1.8.3=hef0d403_0
packaging=23.1=pyhd8ed1ab_0
pandas=1.5.3=py310h2b830bf_1
pango=1.50.14=h9f7e0c6_1
paramiko=3.1.0=pyhd8ed1ab_0
parquet-cpp=1.5.1=2
parso=0.8.3=pyhd8ed1ab_0
partd=1.4.0=pyhd8ed1ab_0
pathable=0.4.3=pyhd8ed1ab_0
pbr=5.11.1=pyhd8ed1ab_0
pcre2=10.40=hb34f9b4_0
pexpect=4.8.0=pyh1a96a4e_2
pickleshare=0.7.5=py_1003
pillow=9.5.0=py310h07496d3_0
pip=23.1.2=pyhd8ed1ab_0
pixman=0.40.0=h27ca646_0
pkgutil-resolve-name=1.3.10=pyhd8ed1ab_0
platformdirs=3.1.1=pyhd8ed1ab_0
pluggy=1.0.0=pyhd8ed1ab_5
pooch=1.7.0=pyha770c72_3
pre-commit=3.3.1=pyha770c72_0
prompt-toolkit=3.0.38=pyha770c72_0
prompt_toolkit=3.0.38=hd8ed1ab_0
psutil=5.9.5=py310h8e9501a_0
pthread-stubs=0.4=h27ca646_1001
ptyprocess=0.7.0=pyhd3deb0d_0
pure_eval=0.2.2=pyhd8ed1ab_0
py-cpuinfo=9.0.0=pyhd8ed1ab_0
py4j=0.10.9.7=pyhd8ed1ab_0
pyarrow=10.0.1=py310h89f3c6b_23_cpu
pyasn1=0.4.8=py_0
pycparser=2.21=pyhd8ed1ab_0
pydantic=1.10.7=py310h8e9501a_0
pygments=2.15.1=pyhd8ed1ab_0
pynacl=1.5.0=py310h8e9501a_2
pyopenssl=23.1.1=pyhd8ed1ab_0
pyparsing=3.0.9=pyhd8ed1ab_0
pyrsistent=0.19.3=py310h8e9501a_0
pysocks=1.7.1=pyha2e5f31_6
pyspark=3.4.0=pyhd8ed1ab_0
pytables=3.8.0=py310h7b1fd78_1
pytest=7.3.1=pyhd8ed1ab_0
pytest-cov=4.0.0=pyhd8ed1ab_0
pytest-rerunfailures=11.1.2=pyhd8ed1ab_0
pytest-timeout=2.1.0=pyhd8ed1ab_0
pytest-xdist=3.2.1=pyhd8ed1ab_0
python=3.10.11=h3ba56d0_0_cpython
python-cityhash=0.4.6=py310h1253130_3
python-dateutil=2.8.2=pyhd8ed1ab_0
python-fastjsonschema=2.16.3=pyhd8ed1ab_0
python-graphviz=0.20.1=pyh22cad53_0
python-jose=3.3.0=pyh6c4a22f_1
python-snappy=0.6.1=py310h7f31a72_0
python-xxhash=3.2.0=py310h8e9501a_0
python_abi=3.10=3_cp310
pytz=2023.3=pyhd8ed1ab_0
pywavelets=1.4.1=py310hf1a086a_0
pywin32-on-windows=0.1.0=pyh1179c8e_3
pyyaml=6.0=py310h8e9501a_5
pyzmq=25.0.2=py310hc407298_0
re2=2023.02.02=hb7217d7_0
readline=8.2=h92ec313_1
regex=2023.5.5=py310h2aa6e3c_0
requests=2.29.0=pyhd8ed1ab_0
responses=0.21.0=pyhd8ed1ab_0
rfc3339-validator=0.1.4=pyhd8ed1ab_0
rsa=4.9=pyhd8ed1ab_0
s3fs=0.4.2=py_0
s3transfer=0.6.1=pyhd8ed1ab_0
sarif-om=1.0.4=pyhd8ed1ab_0
scikit-image=0.20.0=py310h2b830bf_1
scikit-learn=1.2.2=py310h98c5782_1
scipy=1.10.1=py310ha0d8a01_2
setuptools=67.7.2=pyhd8ed1ab_0
six=1.16.0=pyh6c4a22f_0
snappy=1.1.10=h17c5cce_0
sortedcontainers=2.4.0=pyhd8ed1ab_0
sparse=0.14.0=pyhd8ed1ab_0
spectate=1.0.1=pyhd8ed1ab_0
sqlalchemy=1.4.46=py310h8e9501a_0
sshpubkeys=3.3.1=pyhd8ed1ab_0
stack_data=0.6.2=pyhd8ed1ab_0
sympy=1.11.1=pypyh9d50eac_103
tblib=1.7.0=pyhd8ed1ab_0
threadpoolctl=3.1.0=pyh8a188c0_0
tifffile=2023.4.12=pyhd8ed1ab_0
tiledb=2.11.3=h9bd36d0_1
tiledb-py=0.17.3=py310hc1e3be7_0
tk=8.6.12=he1e0b03_0
toml=0.10.2=pyhd8ed1ab_0
tomli=2.0.1=pyhd8ed1ab_0
toolz=0.12.0=pyhd8ed1ab_0
tornado=6.3=py310h8e9501a_0
traitlets=5.9.0=pyhd8ed1ab_0
typing-extensions=4.4.0=hd8ed1ab_0
typing_extensions=4.4.0=pyha770c72_0
tzdata=2023c=h71feb2d_0
ukkonen=1.0.1=py310h2887b22_3
urllib3=1.26.15=pyhd8ed1ab_0
virtualenv=20.21.1=pyhd8ed1ab_0
wcwidth=0.2.6=pyhd8ed1ab_0
websocket-client=1.5.1=pyhd8ed1ab_0
werkzeug=2.3.4=pyhd8ed1ab_0
wheel=0.40.0=pyhd8ed1ab_0
widgetsnbextension=4.0.7=pyhd8ed1ab_0
wrapt=1.15.0=py310h8e9501a_0
xarray=2023.4.2=pyhd8ed1ab_0
xmltodict=0.13.0=pyhd8ed1ab_0
xorg-libxau=1.0.9=h27ca646_0
xorg-libxdmcp=1.1.3=h27ca646_0
xxhash=0.8.1=h1a8c8d9_0
xyzservices=2023.2.0=pyhd8ed1ab_0
xz=5.2.6=h57fd34a_0
yaml=0.2.5=h3422bc3_2
yarl=1.9.1=py310h2aa6e3c_0
zarr=2.14.2=pyhd8ed1ab_0
zeromq=4.3.4=hbdafb3b_1
zfp=1.0.0=hb6e4faa_3
zict=3.0.0=pyhd8ed1ab_0
zipp=3.15.0=pyhd8ed1ab_0
zlib=1.2.13=h03a7124_4
zlib-ng=2.0.7=h1a8c8d9_0
zstd=1.5.2=hf913c23_6
```

cc @martindurant

Contributor guide

Open the contributing guide

Research direction

Start by inspecting the CI test-environment files and checking which s3fs version the CI jobs install, as the issue notes this has not yet been verified. Trace why the environment solver selects s3fs 0.4.2 and confirm the fix makes CI install a current compatible version without the reported performance problem.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.