Regarding Runway, there's this limitation:
By default, each task runs for a maximum of 10 minutes (configurable via spec.timeout in seconds), however the max value for the timeout (i.e., maximum runtime allowed) is 24 hours (86400 seconds).
24 hours is probably enough. But something to keep in mind if we want to do any really big evals in a single execution.
@thomas-schmidt Thanks! I gave Duo a shot at a couple of your suggestions. It implemented one but couldn't quite manage the other, but I've done it now. I also rebased to resolve a merge conflict.
Could you please take another look?
Mark Lapierre (bcafd6ef) at 17 Mar 03:56
refactor: move caching proxy URL logic to Config
... and 87 more commits
Mark Lapierre (d24b7ebe) at 17 Mar 03:53
refactor: move caching proxy URL logic to Config
Mark Lapierre (ab8289e3) at 17 Mar 02:00
refactor: move caching proxy URL logic to Config
Mark Lapierre (86ed44b6) at 17 Mar 01:34
refactor: move caching proxy URL logic to Config
Duo couldn't quite manage the 1st change without a wrapper (which I asked it to remove) so rather than go back and forth slowly here, I finished it locally.
This MR implements the refactoring suggestions from @thomas-schmidt on !4839:
Move _get_anthropic_base_url logic to a method on ConfigDuoWorkflow class: This makes the configuration logic more reusable and keeps all settings in one place, following better object-oriented design principles.
Extract Pyroscope setup logic to a _setup_pyroscope helper function: This improves code organization and readability by separating the Pyroscope setup logic into its own function.
get_anthropic_base_url() method to ConfigDuoWorkflow class in ai_gateway/config.py
_get_anthropic_base_url() function from ai_gateway/models/v2/container.py
_get_anthropic_base_url_from_config() to call the method via dependency injection_setup_pyroscope() helper function in duo_workflow_service/profiling.py
The changes are refactoring only and should not change behavior. Run the existing tests:
poetry run pytest tests/test_config.py::test_duo_workflow_get_anthropic_base_url -v
poetry run pytest tests/models/v2/test_container.py::test_get_anthropic_base_url -vMark Lapierre (df9b15b8) at 17 Mar 01:00
refactor: move caching proxy URL logic to Config
@duo-developer-gitlab-org Could we call import get_anthropic_base_url and call it directly here so we don't need the _get_anthropic_base_url_from_config wrapper?
Could you then also address the test failures in https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/jobs/13518467301
This MR implements the refactoring suggestions from @thomas-schmidt on !4839:
Move _get_anthropic_base_url logic to a method on ConfigDuoWorkflow class: This makes the configuration logic more reusable and keeps all settings in one place, following better object-oriented design principles.
Extract Pyroscope setup logic to a _setup_pyroscope helper function: This improves code organization and readability by separating the Pyroscope setup logic into its own function.
get_anthropic_base_url() method to ConfigDuoWorkflow class in ai_gateway/config.py
_get_anthropic_base_url() function from ai_gateway/models/v2/container.py
_get_anthropic_base_url_from_config() to call the method via dependency injection_setup_pyroscope() helper function in duo_workflow_service/profiling.py
The changes are refactoring only and should not change behavior. Run the existing tests:
poetry run pytest tests/test_config.py::test_duo_workflow_get_anthropic_base_url -v
poetry run pytest tests/models/v2/test_container.py::test_get_anthropic_base_url -v@thomas-schmidt Yup, makes sense!
@duo-developer-gitlab-org Could you implement Thomas' suggestion?
@duo-developer-gitlab-org Could you implement this one too?
@thomas-schmidt Only historical precedent. But I see you're fixing that in !4882. Once that's merged, I could submit a follow-up MR to make these env vars use the config objects. WDYT?
@igor.drozdov I just opened an issue about taking down dws-loadtest
It might be less work to take it down from here (probably in the next day or two, once I get confirmation to go ahead)
But if you want to move it in the meantime, that's fine.
@wortschi WDYT? There's non-trivial effort involved in tearing down dws-loadtest, and in setting it up again if needed. But there's also some effort involved in maintaining it even while it's not in use.