Is there anyway to train "big data" using transformer?

It sames that Transformer reads training data into the Memory. So it easily got OOM Error with "big training data" like 10G (about 50 million text pairs). Is there some solution for this problem?

No files, tests, or entry points are identified in the issue. Begin by tracing how Transformer training data is loaded into memory, then reproduce the out-of-memory case with the described 10G dataset and verify that training can handle about 50 million text pairs without loading the entire dataset at once.

Written by the indexing model from the issue text.