If you start the benchmark with an agent that does not exist, the benchmark should exit with an error. Instead, it just prints but the API server keeps running:
INFO: Started server process [52938]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
[23:37:58] ⚠️ Agent 'debug' not found in registry. Available main.py:63
agents: dict_keys(['stratus', 'autosubmit',
'codex', 'claudecode'])
INFO: Started server process [52938]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:9954 (Press CTRL+C to quit)
If you start the benchmark with an agent that does not exist, the benchmark should exit with an error. Instead, it just prints but the API server keeps running: