Data set which is much bigger than RAM

I'm using Rust, meanwhile I'm new to DataFusion.

I need to repartition big dataset which is hundreds of GB. It is stored on S3 as multiple compressed packet files.
It should be partitioned by the value of a column. Here is what I'm doing

```RUST
// Define the partitioned Listing Table
let listing_options = ListingOptions::new(file_format)
.with_table_partition_cols(part)
.with_target_partitions(1)
.with_file_extension(".ndjson.zst");

ctx.register_listing_table(
"data",
format!("s3://{BUCKET_NAME}/data_lake/data_warehouse"),
listing_options,
Some(schema),
None,
)
.await?;

let df = ctx
.sql(
r#"
SELECT
SUBSTRING("OriginalRequest", 9, 3) as dep, *
FROM data
WHERE
/*partitions predicates here*/

"#,
)
.await?;

let s3 = AmazonS3Builder::new()
.with_bucket_name(save_bucket_name)
.with_region(REGION)
.build()?;

// Register the S3 store in DataFusion context
let path = format!("s3://{save_bucket_name}");
let s3_url = Url::parse(&path).unwrap();
let arc_s3 = Arc::new(s3);
ctx.runtime_env()
.register_object_store(&s3_url, arc_s3.clone());

// Write the data as JSON partitioned by `dep`
let output_path = "s3://my_bucket/output/json/";
//write as JSON to s3

let options = DataFrameWriteOptions::new()
.with_partition_by(vec!["dep".to_string()]);

let mut json_options = JsonOptions::default();
json_options.compression = CompressionTypeVariant::ZSTD;

df
.write_json(&output_path, options, Some(json_options))
.await?;
```

Will it swallow all memory and fail or it will be running in a kind on streaming format?
How could I limit the amount of memory which can be used to run the app inside of the Docker and make sure it would not run out.

No repository file, test, or implementation entry point is named. First determine whether the shown Rust/S3 repartitioned write streams data and how memory limits can be configured in Docker. Done means documenting the observed memory behavior and a supported way to bound usage, if one exists.

Written by the indexing model from the issue text.