← All updates Databricks Update One-Pager Light version
databricks Platform · Updated Jun 23, 2025

Custom rate limits for model serving endpoints

You can now specify custom rate limits for your model serving endpoints using AI Gateway.

Update Open docs ↗release note ↗
What it does

You can now specify custom rate limits for your model serving endpoints using AI Gateway. Provide rate limits for any of the following: Endpoint: Specify the overall rate limit for all traffic passing through the endpoint, regardless of individual or user group limits. User (Default): Specify the rate limit for all users of the endpoint that do not have user specific or group specific rate limits defined.

Why it matters

The new Unity AI Gateway is the enterprise control plane for governing LLM endpoints and coding agents with enhanced features.

Key capabilities
To create an endpoint for
To create an endpoint for external models, complete steps 1 and 2 of Create an external model serving endpoint.
To create an endpoint for
To create an endpoint for provisioned throughput, see Provisioned throughput Foundation Model APIs.
To create an endpoint for
To create an endpoint for a custom model, see Create an endpoint. Endpoint admin operations require CAN MANAGE on that endpoint.…
If the request returns a
If the request returns a 429 or 5xx error on Served entity 1, the request falls back to the next served entity on the endpoint…
Architecture
Databricks Platform → Platform
In short

You can now specify custom rate limits for your model serving endpoints using AI Gateway. Provide rate limits for any of the following: Endpoint: Specify the overall rate limit for all traffic passing through the endpoint, regardless of individual or user…

Read the full documentation ↗
Key takeaway
You can now specify custom rate limits for your model serving endpoints using AI Gateway.
2025-06-23-custom-rate-limits-for-model-serving-endpoints Generated by the Databricks Updates Agent · browse all updates