You have received machine learning model training code, without clear information about the optimal shape...

Start with a smaller shape and monitor the job run metrics and time required to complete the model training. If the compute shape is not fully utilized, tune the model parameters, and rerun the job. Repeat the process until the shape resources are fully utilized.

Start with the strongest compute shape Jobs support and monitor the job run metrics and time required to complete the model training. Tune the model so that it utilizes as much compute resources as possible, even at an increased cost.

Start with a small shape and monitor the utilization metrics and time required to complete the model training. If the compute shape is fully utilized, change to compute that has more resources and rerun the job. Repeat the process until the processing time does not improve.

Start with a random compute shape and monitor the utilization metrics and time required to finish the model training. Perform model training optimization and performance tests in advance to identify the right compute shape before running the model training as a job.