Conversion from one dataset to another that will not fit in memory?

Having found the following description in the documentation, I tried the operation of scanning a dataset larger than memory and writing it to another dataset.

https://arrow.apache.org/docs/python/dataset.html#writing-large-amounts-of-data

> The above examples wrote data from a table. If you are writing a large amount of data you may not be able to load everything into a single in-memory table. Fortunately, the write_dataset() method also accepts an iterable of record batches. This makes it really simple, for example, to repartition a large dataset without loading the entire dataset into memory:

```python
import pyarrow.dataset as ds

input_dataset = ds.dataset("input")
ds.write_dataset(inpute_dataset.scanner(), "output", format="parquet")
```

```r
arrow::open_dataset("input") |>
arrow::write_dataset("output")
```

But both Python and R on Windows crashed due to lack of memory. Am I missing something?
Is there a recommended way to convert one dataset to another without running out of computer memory?

Start with the Python dataset documentation section on writing large amounts of data and reproduce the scanner-to-write_dataset example on Windows; compare it with the R open_dataset |> write_dataset example. Check whether either documented conversion crashes for an input larger than memory, and consider the issue done when the documented operation completes without exhausting memory or the limitation is clearly documented.

Written by the indexing model from the issue text.