dask / dask/dask-cloudprovider
RuntimeWarning: get_log_events rate limit exceeded
- Dominant language
- Python
- Stars
- 147
- Forks
- 119
- PR merge metrics
- No merged PRs in 30d
Description
I'm using [this](https://github.com/rpanai/sagemaker-fargate-test) as reference and whenever I run a distributed computation like
# MCVE
```python
import dask
from dask.distributed import Client, progress
from dask import compute, delayed
from dask_cloudprovider import FargateCluster
%%time
cpu = 1
ram = 2
cluster = FargateCluster(n_workers=1,
image='rpanai/fargate-worker:2020-08-06',
vpc="my_vpc",
subnets=["subnet-1", "subnet-1"],
worker_cpu=int(cpu * 1024),
worker_mem=int(ram * 1024),
cloudwatch_logs_group="my_log_group",
task_role_policies=['arn:aws:iam::aws:policy/AmazonS3FullAccess'],
scheduler_timeout='10 minutes'
)
cluster.adapt(minimum=1,
maximum=40)
client = Client(cluster)
client
```
```python
def fun(fn1):
fn2 = fn1.replace("fldr1", "fldr2")
fn_out = fn1.replace("fldr1", "fldr_out")
df1 = pd.read_parquet(fn1)
df2 = pd.read_parquet(fn2)
df1 = pd.merge(df1, df2)
# stuff
df1.to_parquet(fn_out)
to_process = [delayed(fun)(el)
for el in lst]
out = compute(to_process)
```
# Warning
```
RuntimeWarning: get_log_events rate limit exceeded, retrying after delay.
```
**Environment**:
- Dask version: 2.14
- Python version: 3.6.10
- Operating System: rhel fedora
- Install method (conda, pip, source): conda
**Dockerfile**:
```
FROM continuumio/miniconda3:4.7.12
RUN conda install --yes \
-c conda-forge \
python=3.6.10 \
python-blosc \
cytoolz \
dask==2.14.0 \
dask-ml=1.6.0 \
dask-xgboost=0.1.11 \
msgpack-python=1.0.0 \
nomkl \
numpy==1.19.1 \
pandas==1.1.0 \
numba=0.50.1 \
pyarrow=1.0.0 \
tini==0.18.0 \
pip \
s3fs \
&& conda clean -tipsy \
&& find /opt/conda/ -type f,l -name '*.a' -delete \
&& find /opt/conda/ -type f,l -name '*.pyc' -delete \
&& find /opt/conda/ -type f,l -name '*.js.map' -delete \
&& find /opt/conda/lib/python*/site-packages/bokeh/server/static -type f,l -name '*.js' -not -name '*.min.js' -delete \
&& rm -rf /opt/conda/pkgs
COPY prepare.sh /usr/bin/prepare.sh
RUN mkdir /opt/app
ENTRYPOINT ["tini", "-g", "--", "/usr/bin/prepare.sh"]
```
Contributor guide
Assessment
This issue has not been assessed yet.