Hello! Thanks for flagging this project I wasn't aware of. It sounds interesting and I'll try it soon.

I think it makes sense to keep an eye on alternatives to FastAPI, but:

• Looking at the benchmark I'm not really convinced by their claim: the only relevant graph is the second one, where FastAPI has 6 requests per second versus 8 for LitServe on CPU (the "optimized" version doesn't seem fair as they should do the same with FastAPI then), and 45 vs 57 with a GPU. So it's more about 50% speed up in a benchmark with vested interests and no idea of statistical significance or other nice things. So I'm a bit cautious about this (although it's promising and a decent increase of performance potentially).
• For LLM use cases the underlying framework is rarely the bottleneck
• As you've been experiencing it yourself, there is more than raw speed to consider and the FastAPI ecosystem is convenient and reliable, for things like logging, CORS, middlewares, community, etc... There is a reason why FastAPI is a benchmark that other projects try to challenge regularly.

Cheers!

I mentioned this project in fact more to know if for SimpleAI, it could be an advantage or not.

From what you mention, FastAPi seems to remain the best solution, in the absence of significant performance gains.

Maybe when the library has been tested by the community, its use will become clearer and indispensable.