Python FileBasedCacheManager does not respect PCoder for PCollection being cached
Open
examples
improvement
P3
python
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
FileBasedCacheManager used by Python's InteractiveRunner does not preserve PCoder for elements of a PCollection being cached on disk. I suggest that the cache on-disk format to be changed to TFRecords (which are supported by Beam) and FileBasedCacheManager would store the desired PCoder for cached collections.
Currently, it is not possible to work with dynamically-generated protocol buffer messages in interactive runner mode because of pickling errors.
Imported from Jira [BEAM-6356](https://issues.apache.org/jira/browse/BEAM-6356). Original Jira may contain additional context.
Reported by: leontyev_google.
Contributor guide
Assessment
This issue has not been assessed yet.