GoogleCloudPlatform / GoogleCloudPlatform/training-data-analyst

tfdv_basic_spending.ipynb: version of pyarrow installed incompatible with current version of Apache Beam; breaks generate_statistics_from_dataframe

Open
#1,727 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
8.6k
Forks
6.1k
Avg merge
4h 44m
Merged PRs (30d)
2

Description

On the notebook:

training-data-analyst/courses/machine_learning/deepdive2/production_ml/labs/tfdv_basic_spending.ipynb,

The block where we install tfdv and its dependencies, this installs a version of pyarrow that appears is may not be supported in the current version of Apache Beam (in the GCP environment / context).

This breaks tfdv.generate_statistics_from_dataframe(any_pd_df), as it will return a value error with any dataframe. Following the stack trace, I see that the root cause appears that it may be the error "missing package pyarrow.vendored". This line may be the issue:

`!pip install pyarrow==2.0.0`

It looks like by default, version 5.0.0 is installed (as of time of this writing). This line rolls back version 5.0.0 to 2.0.0. In any case, to troubleshoot the ValueError, when I went back, inserted a new cell, and executed the line below, then I retried tfdv.generate_statistics_from_dataframe(...) ... the value error did not appear any longer, and the issue was apparently fixed:

```
!pip install --upgrade pyarrow
import pyarrow
```

**Recommended fix / patch:**

**replace the line**

`!pip install pyarrow==2.0.0`

**with**
```

!pip install pyarrow==5.0.0
# or other suitable version that doesn't break
# the current version of beam
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.