Is LiteLLM written in Rust? What the codebase and migration tracker show

LiteLLM is not written in Rust. As of November 1, 2026, LiteLLM Rust is a core under construction inside a Python project: GitHub’s language breakdown for the repository shows 84.88% Python and 3.11% Rust, and in the main:latest build only two routes run on Rust. Chat Completions and Messages, the routes chat applications call, still run on Python.

Source context: This analysis draws on LiteLLM’s published benchmark files and Maxim AI’s response. Maxim develops Bifrost, the competing gateway discussed here. No new performance tests were run for this article.

Correction, 5 November 2026: An earlier version omitted the throughput-ceiling file, overstated the consistency of Rust’s memory advantage, and, where discussed, incorrectly treated the approximately 21% CPU figure as unsupported. The ceiling file shows lower peak RSS for Bifrost and supports Rust’s rounded CPU figure.

The question matters because LiteLLM Rust has become a headline. LiteLLM published a benchmark of its Rust gateway against other gateways, and readers who see the numbers can reasonably assume they describe the LiteLLM proxy they deploy. This explainer sets out what the codebase, the Rust gateway documentation and the migration tracker each say, what parts of a gateway those figures cover, and how to read the benchmark in that light. It is written for engineers who run LiteLLM today or are evaluating it, and for anyone who wants to cite the current state accurately.

Cross-check of the published resource files

LiteLLM Rust recorded lower peak RSS in three of four published file sets; Bifrost was lower in the ceiling run at 242.6 MB versus 297.1 MB for Rust. The ceiling results report 0.242614272 GB for Bifrost and 0.29710336 GB for Rust, converted here to decimal MB. The other three sets favor Rust: 21.8 MB versus 199.1 MB in the messages/release capture; 26.1 MB versus about 144 MB in the earlier memory capture; and roughly 35–36 MB versus 282–298 MB in the CPU time series. These are different measurements, not repeated trials of one identical setup.

The ceiling file also records Rust at 21.075% average CPU and 2,814.35 RPS. The separate CPU time series records a much lower average over its own sampling window. The figures must retain their file and workload context.

The Bifrost post identifies v1.6.4 as the tested build and infers default-on logging from v2.4.0 code; it explicitly says that behavior was not yet confirmed on v1.6.4. Neither the blog nor the benchmark files quantify logging’s share of the memory or CPU difference.

LiteLLM Rust in one table

The short answer is that three separate measurements describe the LiteLLM Rust effort, and each measures something different: bytes of code, routes served, and engineering work completed. None of them says LiteLLM has been rewritten in Rust, and none of them is a measure of how much production traffic Rust handles.

Each row is taken from the project’s own pages, as of November 1, 2026. The sections below take them in turn, then put them together.

What the GitHub language breakdown measures

GitHub’s language breakdown is a byte count, not a feature map. As of November 1, 2026, the breakdown for BerriAI/litellm shows 84.88% Python, 3.11% Rust and 12.01% other languages, measured by source-code bytes. That includes tests and supporting code, so it cannot say what share of requests or features runs on Rust.

The figure comes from the repository’s language endpoint on the GitHub API, the same data behind the coloured bar on the BerriAI/litellm repository page. GitHub classifies each file by language and sums the bytes. Three properties of that method shape how the number should be read.

First, bytes are not behaviour. A small Rust module that handles a hot path could matter more to latency than a large Python module that handles configuration. The reverse is also possible: a large body of Rust code can sit unused if the proxy does not route requests to it. The byte count does not distinguish between the two.

Second, the count includes everything in the repository. Test suites, fixtures, build tooling and documentation helpers all add bytes. The 3.11% does not mean 3.11% of features or traffic run on Rust.

Third, the number moves. Every merge changes the byte totals, which is why any citation of it should carry a date. The figures here are the ones recorded on November 1, 2026.

What the breakdown does establish is direction and scale. A project in which Python makes up 84.88% of the source is a Python project. The Rust code is real, but by this measure it is a small part of the whole. To learn what that Rust code actually does at runtime, the language bar is the wrong instrument. The routing documentation and the migration tracker are the right ones.

The parts of an AI gateway, and why each one matters

An AI gateway is several subsystems behind one HTTP endpoint: a server, authentication, a router, budgets and rate limits, provider translation, caching, cost calculation, logging, and the database and configuration that back them. A Rust port is complete only when a request can pass through all of the parts it needs without returning to Python.

That framing is useful because LiteLLM’s tracker reports progress component by component, and the components are not equal. Some do work once per request in memory. Others read or write shared state on every call. The site’s AI gateway explainer walks through the request path in detail; the summary below is the version needed to read the LiteLLM figures.

Two groups stand out. Provider translation and the HTTP layer are stateless: they transform bytes and pass them on. Porting them changes how the bytes are moved, not what the gateway enforces. The router, budgets, rate limits, spend tracking and database access are stateful: they coordinate across requests and, in a multi-instance deployment, across machines. These are the parts that turn a forwarding proxy into a gateway that can enforce a team’s budget or fail over from one provider to another.

The distinction explains why a benchmark of forwarding overhead and a description of production behaviour can diverge. A gateway configured to translate and forward, with no budgets checked and nothing logged, does less work than a governed production gateway does on the same request.

Which LiteLLM routes run on Rust in main:latest

In main:latest, LiteLLM’s Python proxy serves two routes through Rust: /ocr for every provider and /audio/transcriptions for Bedrock. Every other route runs on Python, including Messages and Chat Completions. The per-model rust: true flag in configuration currently has no effect, according to LiteLLM’s own Rust gateway documentation.

The Rust gateway documentation carries this as a warning at the top of the page. It is the most direct answer to the question of what runs on Rust, because it describes the build users pull, not a roadmap.

Three details are easy to miss.

The flag does not switch anything on. LiteLLM’s configuration has a per-model rust: true setting. The warning states that, in main:latest, the flag currently has no effect. A team that has set it on a Chat Completions model is running that model on Python.

The page contains older descriptions beneath the warning. The current Rust docs retain older coverage descriptions beneath the warning, so a reader who skips the warning and reads the coverage lists further down can come away with a broader picture than main:latest delivers. The practical consequence is that deployment claims need a specific version. A statement such as “LiteLLM runs Chat Completions on Rust” is only checkable against a named release.

The two Rust routes are not the chat routes. OCR and Bedrock audio transcription are real workloads, but chat applications call Chat Completions or Messages. For a team whose traffic is chat, the share of its requests served by Rust in main:latest is zero.

Figure 1: In main:latest, the route decides whether Rust is involved, and for chat traffic it is not.

What the migration tracker’s 24% counts

LiteLLM’s migration tracker reports 24% of Major APIs built as of v1.107.0-rc.2, which is 35.4 of 146 weighted engineering points. The percentage counts released engineering work, not traffic served by Rust. The router, database access, budgets and rate limits, and database-backed configuration are each at 0%.

The migration tracker is the project’s own progress report, and it is the most granular of the three sources. Its state falls into three groups.

Reading this against the component table above gives a clear picture. The stateless foundations are in place: an HTTP layer, streaming, tracing, the model catalog and configuration parsing. Translation and several request-time features are partly built. The stateful components that make LiteLLM a governed gateway rather than a translator, namely the router, budgets, rate limits and database access, are all at 0% on the tracker’s scale.

The necessary conclusion about the partial components is that their presence does not establish an end-to-end Rust request path. A request that needs routing, a budget check and a spend write still needs Python for each of those steps, even if its translation could in principle run in Rust.

Why weighted points are not a traffic share

The 24% is a weighted sum: 35.4 released points out of 146 weighted engineering points across the Major APIs. That is a reasonable way to report a migration’s progress to contributors, and it is a poor way to answer “how much of my traffic runs on Rust”.

A worked example shows the gap. A team whose traffic is entirely Chat Completions, routed with fallbacks and governed by per-key budgets, touches the router, budgets and rate limits on every request. All three are at 0% on the tracker, and the Chat Completions route itself runs on Python in main:latest. For that team, 24% of Major APIs translates to none of its requests on Rust today. A team running OCR jobs through /ocr is in a different position: that route is on Rust in main:latest, though the proxy around it is still Python.

Figure 2: The stateless foundations are built; the stateful parts that make a proxy a gateway are at 0%.

Why the router, budgets and rate limits decide the answer

The router, budgets, rate limits and database are where a production gateway does its per-request coordination. Until those run in Rust, any request that uses them passes through Python. They also sit outside what a forwarding-only benchmark measures, which is why their 0% status matters for reading LiteLLM Rust performance claims.

Each of these components does work that a translation layer does not.

The router

A router chooses which deployment of a model handles a request. In a typical configuration, several deployments sit behind one model name: different providers, regions or API keys. The router tracks which ones are healthy, places failing deployments in cooldown, retries transient errors and falls back to another deployment when one fails. That logic runs on every request and holds state between requests. The tracker lists router migration at 0%, and routing remains among the responsibilities the Python proxy still owns.

Budgets and rate limits

Budgets and rate limits are checked before the provider is called. A gateway that enforces a monthly spend limit for a team, or a request-per-minute cap for a key, must read the current count, compare it to the limit and update it, consistently across every instance of the gateway. That shared, consistent state is something a stateless translation step never needs. The tracker lists budgets and rate limits at 0%.

Database access and database-backed configuration

In a gateway of this kind, keys, budgets and, where configuration is database-backed, the model list itself live in a database. Database access and database-backed configuration are both at 0% on the tracker. A Rust request path that needs to look up a key or load a model stored in the database cannot do so in Rust yet.

Spend tracking and logging

Spend tracking and logging happen after the response. The tracker shows cost calculation and logging partly implemented, and logging, callbacks and spend tracking remain among the jobs the Python proxy still owns. These are also the features that LiteLLM’s benchmark disabled, as the next section shows.

The pattern across all four is the same. The components at 0% on the tracker are ones a team relies on when it uses LiteLLM as a gateway rather than as a format translator. That does not make the Rust work unimportant. It means that, for a governed request, the Rust core cannot yet carry the request from end to end, which is the point made above about the partial components.

What LiteLLM’s Rust benchmark measured

LiteLLM’s benchmark measured a narrower Rust beta, not the full LiteLLM product rewritten in Rust. Its blog says every gateway ran “with no logging callbacks, spend tracking, or persistence enabled”, and calls the Rust gateway an early beta in which “streaming and the full feature surface are still landing”. The results describe forwarding overhead under that configuration only.

The benchmark blog post and the ai-gateway-bench results are explicit about scope, and the blog itself says “It is not a full-feature comparison”. The consequence is that those results cannot establish the performance of the complete production feature set.

Put next to the tracker, the scope is easy to state. The benchmark disabled logging callbacks, spend tracking and persistence. The tracker shows the router, database access, budgets and rate limits at 0%. Neither the features the benchmark switched off nor the components still missing from the Rust core are reflected in the Rust results.

Figure 3: The benchmark timed the forwarding path; the stateful work a production gateway adds was outside it.

Versions and configuration

The comparison set the Rust beta, which has no streaming yet, against Bifrost v1.6.4, an older Bifrost build. The Bifrost version appears in the blog and in the Bifrost gateway README in the benchmark repo, not on LiteLLM’s homepage.

There is also a configuration gap. LiteLLM’s blog says every gateway ran with logging and persistence off, but the published Bifrost config never sets enable_logging or logs_store. In Bifrost, enable_logging is the master toggle for request tracing, documented on the built-in observability page, and it defaults to true when unset and creates a default SQLite log store. That finding is an inference from v2.4.0 code and not yet confirmed on v1.6.4. If it holds, Bifrost’s results include log buffering and SQLite writes, which the v1.6.4 log store writer configuration shows being batched and flushed. A controlled rerun is needed to measure that contribution.

Method

The files publish single runs. The blog describes “single-host, per-scenario runs (for example n=5000 on the overhead panel) without repeated-trial error bars” and asks readers to “treat them as order-of-magnitude differences”. No machine type, core count or worker count is given for any gateway. Those limits apply to every number in the sweep, including the ones that favour Rust. The site’s editorial on vendor benchmarks and their footnotes covers why those disclosures matter in general.

The Python proxy numbers in the same sweep

The benchmark’s own sweep includes LiteLLM’s Python proxy, and it shows the gap between the Rust beta and the Python proxy that serves Chat Completions in main:latest. At concurrency 64, the Python proxy adds 546.04 ms of p99 latency and reaches 179.8 requests per second, against 40.28 ms and 2,722.2 requests per second for the Rust beta.

Those figures come from latency_vs_concurrency.csv in LiteLLM’s benchmark repo. All error counts in the file are zero.

Three readings follow from the table.

The Python proxy is the outlier. At concurrency 64 it adds 546 ms of p99, about 13.6x the Rust port, and reaches about 180 requests per second, 6% of the direct baseline. At concurrency 16 it already adds 251.34 ms.

The Rust beta and Bifrost are close. Every throughput gap between them is under 1%: at concurrency 64, Bifrost keeps 95.2% of direct throughput and Rust 95.3%. On added p99, Rust is lower at concurrency 16 (+0.33 ms against +0.89 ms, a 0.56 ms difference), effectively tied at 4 (a 0.02 ms gap), and higher at 1 and 64. With one run per cell and no confidence intervals, the significance of these small gaps is unknown, and the CSV shows similar throughput for LiteLLM Rust and Bifrost in the tested configurations. GenAI Brief’s explainer on throughput vs latency in gateway benchmarks explains why the two gateways tie on requests per second.

Rust does lead elsewhere in the files. Three results favour Rust. Rust records lower CPU: a mean of 0.66% against Bifrost’s 2.59%, with peaks of 15.9% and 79.5%. Rust records lower peak RSS in three of four published file sets, and the ceiling file instead records Rust at 297.1 MB and Bifrost at 242.6 MB; GenAI Brief’s explainer on LiteLLM’s peak RSS memory figures covers the three non-ceiling memory pairs in detail. At concurrency 256, Rust records lower latency (p50 55.5 ms against 89.3 ms, p99 2,667.8 ms against 5,421.3 ms), though that is a single exploratory run of 5,000 requests per gateway, with no RPS field, which the repo excluded from the sweep because the direct baseline itself exceeded the benchmark’s latency threshold. These results describe the tested configurations, with the differences in logging noted above.

Why the Python column is the one to read

For a team on main:latest, the relevant column depends on the route. Chat Completions and Messages run on Python, so the Python proxy’s behaviour is what those requests get, whatever the per-model flag says. The Rust column describes a beta configuration with logging, spend tracking and persistence off, on a path that main:latest does not use for chat. The Rust column measures forwarding overhead for that beta. It is not a picture of the LiteLLM proxy in main:latest today, and it does not measure the router, budgets or database work that the Python proxy still handles.

Put precisely, the benchmark measures a narrower Rust beta, not the full LiteLLM product rewritten in Rust.

What teams running LiteLLM today actually run

A team running LiteLLM from main:latest runs a Python proxy. Authentication, routing, logging, callbacks and spend tracking are handled in Python, and Chat Completions and Messages are served on Python. Only /ocr and Bedrock /audio/transcriptions reach the Rust core. Setting rust: true on a model does not change that in this build.

That translates into a few concrete situations.

A team serving chat through LiteLLM

A product team sends Chat Completions traffic through LiteLLM with fallbacks between two providers and per-key budgets. Every request uses the router, the budget check and spend tracking. In main:latest all of that runs on Python, and so does the Chat Completions route. The Python column in the sweep is the closer proxy for its overhead, with the caveat that the benchmark ran with logging and spend tracking off, so a governed deployment does more work than the benchmarked one.

A team that has set rust true

A platform team read the earlier coverage descriptions in the Rust docs and added rust: true to its chat models. In main:latest, LiteLLM’s documentation says the flag has no effect. The team’s traffic is on Python. Before attributing any latency change to Rust, it should check the warning on the Rust gateway page for the exact version it deploys.

A team running OCR or Bedrock transcription

A document-processing team sends /ocr requests, or a voice team sends Bedrock /audio/transcriptions requests. These routes run on Rust in main:latest. The proxy that receives the request is still the Python proxy, and the Python proxy still owns authentication, logging, callbacks and spend tracking.

A team evaluating LiteLLM against other gateways

A team comparing gateways should benchmark the configuration it will run: the routes it uses, with logging, budgets and rate limits configured as in production, on a named release. The LiteLLM Rust sweep can inform expectations for a future forwarding path, but it is not a substitute for a test of the current proxy. The site’s LiteLLM alternatives guide and its comparison of five LLM gateways cover the wider options.

Checks worth running before citing LiteLLM Rust figures

• Name the version. Deployment claims need a specific version. Cite the release, not “LiteLLM”.
• Name the route. “Rust” in main:latest means /ocr and Bedrock /audio/transcriptions. Chat Completions and Messages are Python.
• Separate bytes, routes and points. 3.11% is source-code bytes, 24% is weighted engineering points, and neither is a traffic share.
• Match the benchmark’s configuration to yours. The Rust results were taken with logging callbacks, spend tracking and persistence off.
• Date the citation. The language bar, the docs and the tracker all change. The figures in this article are as of November 1, 2026.

How to read claims about LiteLLM and Rust

The accurate summary, as of November 1, 2026, is that LiteLLM is a Python gateway with a partial Rust core. The codebase is 3.11% Rust by bytes, the tracker shows 24% of Major APIs built, two routes run on Rust in main:latest, and the stateful gateway components sit at 0% on the tracker.

That is a reasonable state for a migration in progress, and the project documents it openly: the warning on the Rust gateway page and the component-level tracker are the sources this article relies on. The confusion arises when the benchmark, which measured a forwarding-only beta, is read as a description of the product. The benchmark answers “how much overhead does the Rust forwarding path add, with logging and persistence off”. It does not answer “how fast is LiteLLM”, and it does not answer “how fast is the LiteLLM a team deploys from main:latest today”.

The figures worth watching are the ones that move the answer. The first is the route list in the Rust gateway warning: when Chat Completions or Messages appear on it for a named release, the share of chat traffic on Rust changes. The second is the router, database, budgets, rate limits and configuration rows on the tracker, all at 0% today. Until those move, a governed LiteLLM request still depends on Python for its routing and its spend controls. The third is a like-for-like benchmark: Maxim’s proposed Part 2 follow-up will run both gateways with logging off and on, on a current Bifrost build, with repeated trials, error bars and published hardware.

Until then, the defensible way to cite LiteLLM Rust is narrow and dated: a partial migration, a beta benchmarked with features off, and a Python proxy serving Chat Completions and Messages. Readers who want the broader context on what a gateway is for can start with the site’s explainer on how an AI gateway handles a request.

Source: Debunking LiteLLM Rust Benchmarking: Part 1, by Akshay Deo of Maxim AI, which builds the Bifrost gateway.