Charges Spark : mesurer avant de changer la capacité
Comparez un traitement par lots selon sa durée, ses résultats, ses ressources et les charges concurrentes.
Article de référence en anglais
Define a representative job
Choose one Spark workload with known input and expected output. Record its input snapshot, partitioning, transformation steps, runtime environment, and current schedule. Keep the source data and test conditions stable while comparing changes. A smaller runtime is not useful if a modification drops records or changes the meaning of an aggregate.
Work through a controlled comparison
Suppose an illustrative import reads 100 files and writes a daily customer summary. First record row counts, distinct customer keys, and the sum of the relevant amount field. Then change one implementation choice, such as avoiding an unnecessary full-data scan. Compare both resource consumption and correctness. Do not change cluster configuration, code, and input volume simultaneously if you need to identify the cause of a difference.
Check the schedule as well as the job
Background consumption can compete with other workloads assigned to the capacity. Moving a job does not guarantee that its effect disappears, because capacity accounting and smoothing affect when consumption is evaluated. Check official workload guidance and observe the actual capacity metrics around the reporting period instead of assuming an off-hours start makes the workload free of contention.
Release checklist
- Preserve a reproducible input sample and output checks.
- Record runtime and consumed resources over repeated runs.
- Include timeouts, retries, skewed data, and a failed-write case.
- Compare the job alongside the intended reporting workload.
- Verify service-specific limits before increasing compute.
If a change improves only the synthetic sample, retain that limitation in the decision record. Use the site's Spark estimator to explore assumptions, then validate the selected configuration with a pilot. The estimator does not inspect your notebook, verify correctness, or establish a throughput guarantee.