When Mount prefix matching hides a route

A customer reported that calls to a newly created agent always returned 404.

POST /agent/api/v2/a1/main
-> {"detail":"Not Found"}

Everything we could check looked fine. The console showed the agent as deployed, the API access toggle was on, and the registration API had returned 200. Other agents on the same site worked.

POST /agent/api/a10/test   -> 200

It wasn't the gateway

With a 404 you usually look at the edge first: a bad routing rule, or an ingress that can't resolve the path.

But the request was in the pod log with a response time of 24ms, so it had passed the edge and reached the application. And the response body, {"detail":"Not Found"}, was FastAPI's default 404 format, different from the response envelope we use.

The app had received the request and answered that it couldn't find the route, and that route was registered.

Mounts match by prefix

For every agent a user registers, the engine builds a separate FastAPI sub-application and attaches it to the root app under the agent's alias.

app.mount(f"/{agent_alias}", subapp)

The site in question had two agents with these aliases.

Starlette's router walks the registered routes in order and uses the first match, and Mount matches by prefix.

The deploy succeeded and the route exists; it's just shadowed by an earlier mount and never called.

Reproducing it with the mount order swapped

I reproduced it in the project virtualenv.

from fastapi import FastAPI
from fastapi.testclient import TestClient
 
sa = FastAPI()
@sa.post("/a10/test")
def _a(): return {"ok": 1}
 
sb = FastAPI()
@sb.post("/a1/main")
def _b(): return {"ok": 2}
 
app = FastAPI()
app.mount("/api", sa)      # shorter path first
app.mount("/api/v2", sb)
 
c = TestClient(app)
c.post("/api/a10/test")    # 200
c.post("/api/v2/a1/main")  # 404

The result depends on mount order, mount order comes from the order in which a restarting pod restores deployment state, and that order isn't guaranteed. So a pod restart can make the symptom disappear, and the next one can bring it back.

Sort the mounts, or block the alias

One way out was to reject aliases containing a slash at registration. The other was to sort mounts by descending path length so that longer paths match first.

Sorting looked more fundamental, since it's safe for any alias that might arrive. But agents are added and removed at runtime all the time, so every change would mean re-sorting the root app's route list, and a request landing between unmount and remount would race with it. We'd be carrying code that shuffles the routing table while serving traffic, for good.

So I checked what sorting would actually prevent.

app.mount("/api",   ...)   # first
app.mount("/apiv2", ...)   # second
-> /api/x    200
-> /apiv2/y  200

Starlette's Mount matches on segment boundaries, so /api doesn't intercept /apiv2/y. Prefix here turned out to mean path prefix, not string prefix.

So a collision only happens when one alias is a path prefix of another, and that's only possible if an alias contains a slash. Restrict aliases to a single segment and no collision can happen, and the sorting becomes code that never changes anything.

I also confirmed that no alias in production contained a slash; the offending api/v2 had been created for a test. We left sorting out and made registration reject aliases with a slash.

Why it had to be caught at registration

Until then the alias field was declared as a plain string with no format check. The only thing registration verified was whether it matched the agent's own previous alias, so an alias with a slash went straight through.

What made it worse was that the bad input was recorded as a success. The registration API returned 200, the console showed deployed, and the route was really created, so every signal a user could see said fine. The sign that something was wrong came hours later as a 404 on a different agent, somewhere that looked unrelated to the cause.

Now that registration blocks it, no new collision can happen. But the console's "deployed" only goes as far as "the file was written and the route was registered", so nobody checks that the route can actually be reached. A post-deploy check that calls the agent would have surfaced this right after registration.

Because the defect depends on order, it's also hard to catch with a test ahead of time. The reproduction above was written after the cause was known, and catching it without knowing the cause would have meant thinking up a test that registers two agents in one specific order. That Mount matches by prefix and the router takes the first match is in the documentation, but it was only after reproducing it that I connected that to "allowing a slash in an alias makes some agent permanently unreachable".

End