An ML engineer wants to run a training job on Amazon SageMaker AI by using...

Large Parquet files can cause out-of-memory (OOM) issues during training if individual files exceed the memory capacity of the training instances. AWS documentation recommends repartitioning large datasets into smaller files to enable efficient streaming and parallel loading.

By using Apache Spark on Amazon EMR to repartition the Parquet files, the ML engineer can split the dataset into multiple smaller files that can be read incrementally during training. This approach avoids loading a single large file into memory and improves data parallelism.

Attaching larger EBS volumes increases storage capacity but does not solve memory constraints. Switching to memory-optimized instances increases cost and is not necessary when the dataset can be restructured. SageMaker distributed data parallelism focuses on model parameter synchronization across GPUs, not dataset file size.

AWS best practices explicitly recommend partitioning large Parquet datasets to improve memory efficiency during training.

Therefore, Option B is the correct and AWS-aligned solution.