databricks Platform · Updated Aug 14, 2025

Token-based rate limits now available on AI Gateway

You can now configure token based rate limits on your model serving endpoints.

What it does

You can now configure token based rate limits on your model serving endpoints. Try the new Unity AI Gateway Beta A new Unity AI Gateway experience is available in Beta. The new Unity AI Gateway is the enterprise control plane for governing LLM endpoints and coding agents with enhanced features.

Why it matters

The new Unity AI Gateway is the enterprise control plane for governing LLM endpoints and coding agents with enhanced features.

Key capabilities

To create an endpoint for

To create an endpoint for external models, complete steps 1 and 2 of Create an external model serving endpoint.

To create an endpoint for

To create an endpoint for provisioned throughput, see Provisioned throughput Foundation Model APIs.

To create an endpoint for

To create an endpoint for a custom model, see Create an endpoint. Endpoint admin operations require CAN MANAGE on that endpoint.…

If the request returns a

If the request returns a 429 or 5xx error on Served entity 1, the request falls back to the next served entity on the endpoint…

Architecture

Databricks Platform → Platform

In short

You can now configure token based rate limits on your model serving endpoints. Try the new Unity AI Gateway Beta A new Unity AI Gateway experience is available in Beta.

Read the full documentation ↗

Key takeaway

You can now configure token based rate limits on your model serving endpoints.

2025-08-14-token-based-rate-limits-now-available-on-ai-gateway Generated by the Databricks Updates Agent · browse all updates