Text-to-Speech Scaling Method for Amazon SageMaker AI

Amazon Web Services (AWS) has implemented a new measurement methodology that eliminates uncertainties in auto-scaling configurations for text-to-speech (TTS) models hosted on Amazon SageMaker AI. Aiming to prevent distortions caused by standard policies, this approach relies on empirical measurements taken from live endpoints.

Standard auto-scaling policies are often based on estimated target values, which can lead to idle GPU resources or service level objective violations during traffic spikes. The newly introduced method solves this issue by basing the deployment of the Kokoro-82M model as a custom container on a SageMaker real-time endpoint.

The system derives threshold values from two custom measurements performed on the live endpoint instead of default metrics: Q_max, which indicates the maximum concurrency a single instance can sustain without violating targets, and T_total, which represents the total time required for a scaling operation. In tests conducted for the Kokoro-82M model, the Q_max value was determined to be 50 concurrent requests, and the T_total duration was approximately 260.2 seconds.

The deployment process is managed via AWS Cloud Development Kit (AWS CDK) stacks, which include an independent control mechanism that ensures scaling resources are not synthesized unless a verified measurement plan exists. This prevents unmeasured and predictive configurations from reaching production environments.