Skip to content

Commit c68478f

Browse files
li-boxuanli-boxuan
authored
Customize LLM config per agent (OpenHands#2756)
Currently, OpenDevin uses a global singleton LLM config and a global singleton agent config. This PR allows customers to configure an LLM config for each agent. A hypothetically useful scenario is to use a cheaper LLM for repo exploration / code search, and a more powerful LLM to actually do the problem solving (CodeActAgent). Partially solves OpenHands#2075 (web GUI improvement is not the goal of this PR)
1 parent 23e2d01 commit c68478f

35 files changed

Lines changed: 522 additions & 227 deletions

File tree

‎.github/workflows/dummy-agent-test.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ jobs:
3131
- name: Run tests
3232
run: |
3333
set -e
34-
poetry run python opendevin/core/main.py -t "do a flip" -m ollama/not-a-model -d ./workspace/ -c DummyAgent
34+
poetry run python opendevin/core/main.py -t "do a flip" -d ./workspace/ -c DummyAgent
3535
- name: Check exit code
3636
run: |
3737
if [ $? -ne 0 ]; then

‎agenthub/codeact_agent/codeact_agent.py‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
)
99
from opendevin.controller.agent import Agent
1010
from opendevin.controller.state.state import State
11+
from opendevin.core.config import config
1112
from opendevin.events.action import (
1213
Action,
1314
AgentDelegateAction,
@@ -60,8 +61,11 @@ def get_action_message(action: Action) -> dict[str, str] | None:
6061

6162

6263
def get_observation_message(obs) -> dict[str, str] | None:
64+
max_message_chars = config.get_llm_config_from_agent(
65+
'CodeActAgent'
66+
).max_message_chars
6367
if isinstance(obs, CmdOutputObservation):
64-
content = 'OBSERVATION:\n' + truncate_content(obs.content)
68+
content = 'OBSERVATION:\n' + truncate_content(obs.content, max_message_chars)
6569
content += (
6670
f'\n[Command {obs.command_id} finished with exit code {obs.exit_code}]'
6771
)
@@ -76,10 +80,12 @@ def get_observation_message(obs) -> dict[str, str] | None:
7680
'![image](data:image/png;base64, ...) already displayed to user'
7781
)
7882
content = '\n'.join(splitted)
79-
content = truncate_content(content)
83+
content = truncate_content(content, max_message_chars)
8084
return {'role': 'user', 'content': content}
8185
elif isinstance(obs, AgentDelegateObservation):
82-
content = 'OBSERVATION:\n' + truncate_content(str(obs.outputs))
86+
content = 'OBSERVATION:\n' + truncate_content(
87+
str(obs.outputs), max_message_chars
88+
)
8389
return {'role': 'user', 'content': content}
8490
return None
8591

‎agenthub/codeact_swe_agent/codeact_swe_agent.py‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
from agenthub.codeact_swe_agent.response_parser import CodeActSWEResponseParser
88
from opendevin.controller.agent import Agent
99
from opendevin.controller.state.state import State
10+
from opendevin.core.config import config
1011
from opendevin.events.action import (
1112
Action,
1213
AgentFinishAction,
@@ -52,8 +53,11 @@ def get_action_message(action: Action) -> dict[str, str] | None:
5253

5354

5455
def get_observation_message(obs) -> dict[str, str] | None:
56+
max_message_chars = config.get_llm_config_from_agent(
57+
'CodeActSWEAgent'
58+
).max_message_chars
5559
if isinstance(obs, CmdOutputObservation):
56-
content = 'OBSERVATION:\n' + truncate_content(obs.content)
60+
content = 'OBSERVATION:\n' + truncate_content(obs.content, max_message_chars)
5761
content += (
5862
f'\n[Command {obs.command_id} finished with exit code {obs.exit_code}]'
5963
)
@@ -68,7 +72,7 @@ def get_observation_message(obs) -> dict[str, str] | None:
6872
'![image](data:image/png;base64, ...) already displayed to user'
6973
)
7074
content = '\n'.join(splitted)
71-
content = truncate_content(content)
75+
content = truncate_content(content, max_message_chars)
7276
return {'role': 'user', 'content': content}
7377
return None
7478

‎agenthub/micro/agent.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
from opendevin.controller.agent import Agent
44
from opendevin.controller.state.state import State
5+
from opendevin.core.config import config
56
from opendevin.core.utils import json
67
from opendevin.events.action import Action
78
from opendevin.events.serialization.action import action_from_dict
@@ -32,14 +33,17 @@ def history_to_json(history: ShortTermHistory, max_events=20, **kwargs):
3233
"""
3334
Serialize and simplify history to str format
3435
"""
36+
# TODO: get agent specific llm config
37+
llm_config = config.get_llm_config()
38+
max_message_chars = llm_config.max_message_chars
3539

3640
processed_history = []
3741
event_count = 0
3842

3943
for event in history.get_events(reverse=True):
4044
if event_count >= max_events:
4145
break
42-
processed_history.append(event_to_memory(event))
46+
processed_history.append(event_to_memory(event, max_message_chars))
4347
event_count += 1
4448

4549
# history is in reverse order, let's fix it

‎agenthub/monologue_agent/agent.py‎

Lines changed: 17 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@
2929
from opendevin.memory.condenser import MemoryCondenser
3030
from opendevin.runtime.tools import RuntimeTool
3131

32-
if config.agent.memory_enabled:
32+
if config.get_agent_config('MonologueAgent').memory_enabled:
3333
from opendevin.memory.memory import LongTermMemory
3434

3535

@@ -78,7 +78,7 @@ def _initialize(self, task: str):
7878
raise AgentNoInstructionError()
7979

8080
self.initial_thoughts = []
81-
if config.agent.memory_enabled:
81+
if config.get_agent_config('MonologueAgent').memory_enabled:
8282
self.memory = LongTermMemory()
8383
else:
8484
self.memory = None
@@ -89,6 +89,9 @@ def _initialize(self, task: str):
8989
self._initialized = True
9090

9191
def _add_initial_thoughts(self, task):
92+
max_message_chars = config.get_llm_config_from_agent(
93+
'MonologueAgent'
94+
).max_message_chars
9295
previous_action = ''
9396
for thought in INITIAL_THOUGHTS:
9497
thought = thought.replace('$TASK', task)
@@ -106,7 +109,9 @@ def _add_initial_thoughts(self, task):
106109
observation = BrowserOutputObservation(
107110
content=thought, url='', screenshot=''
108111
)
109-
self.initial_thoughts.append(event_to_memory(observation))
112+
self.initial_thoughts.append(
113+
event_to_memory(observation, max_message_chars)
114+
)
110115
previous_action = ''
111116
else:
112117
action: Action = NullAction()
@@ -133,7 +138,7 @@ def _add_initial_thoughts(self, task):
133138
previous_action = ActionType.BROWSE
134139
else:
135140
action = MessageAction(thought)
136-
self.initial_thoughts.append(event_to_memory(action))
141+
self.initial_thoughts.append(event_to_memory(action, max_message_chars))
137142

138143
def step(self, state: State) -> Action:
139144
"""
@@ -145,15 +150,17 @@ def step(self, state: State) -> Action:
145150
Returns:
146151
- Action: The next action to take based on LLM response
147152
"""
148-
153+
max_message_chars = config.get_llm_config_from_agent(
154+
'MonologueAgent'
155+
).max_message_chars
149156
goal = state.get_current_user_intent()
150157
self._initialize(goal)
151158

152159
recent_events: list[dict[str, str]] = []
153160

154161
# add the events from state.history
155162
for event in state.history.get_events():
156-
recent_events.append(event_to_memory(event))
163+
recent_events.append(event_to_memory(event, max_message_chars))
157164

158165
# add the last messages to long term memory
159166
if self.memory is not None:
@@ -163,9 +170,11 @@ def step(self, state: State) -> Action:
163170
# this should still work
164171
# we will need to do this differently: find out if there really is an action or an observation in this step
165172
if last_action:
166-
self.memory.add_event(event_to_memory(last_action))
173+
self.memory.add_event(event_to_memory(last_action, max_message_chars))
167174
if last_observation:
168-
self.memory.add_event(event_to_memory(last_observation))
175+
self.memory.add_event(
176+
event_to_memory(last_observation, max_message_chars)
177+
)
169178

170179
# the action prompt with initial thoughts and recent events
171180
prompt = prompts.get_request_action_prompt(

‎agenthub/planner_agent/prompt.py‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
from opendevin.controller.state.state import State
2+
from opendevin.core.config import config
23
from opendevin.core.logger import opendevin_logger as logger
34
from opendevin.core.schema import ActionType
45
from opendevin.core.utils import json
@@ -128,6 +129,9 @@ def get_prompt(state: State) -> str:
128129
Returns:
129130
- str: The formatted string prompt with historical values
130131
"""
132+
max_message_chars = config.get_llm_config_from_agent(
133+
'PlannerAgent'
134+
).max_message_chars
131135

132136
# the plan
133137
plan_str = json.dumps(state.root_task.to_dict(), indent=2)
@@ -142,7 +146,7 @@ def get_prompt(state: State) -> str:
142146
break
143147
if latest_action == NullAction() and isinstance(event, Action):
144148
latest_action = event
145-
history_dicts.append(event_to_memory(event))
149+
history_dicts.append(event_to_memory(event, max_message_chars))
146150

147151
# history_dicts is in reverse order, lets fix it
148152
history_dicts.reverse()
@@ -160,7 +164,7 @@ def get_prompt(state: State) -> str:
160164
plan_status = "You're not currently working on any tasks. Your next action MUST be to mark a task as in_progress."
161165

162166
# the hint, based on the last action
163-
hint = get_hint(event_to_memory(latest_action).get('action', ''))
167+
hint = get_hint(event_to_memory(latest_action, max_message_chars).get('action', ''))
164168
logger.info('HINT:\n' + hint, extra={'msg_type': 'DETAIL'})
165169

166170
# the last relevant user message (the task)

‎config.template.toml‎

Lines changed: 23 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -79,8 +79,12 @@ persist_sandbox = false
7979
# Use host network
8080
#use_host_network = false
8181

82+
# Name of the default agent
83+
#default_agent = "CodeActAgent"
84+
8285
#################################### LLM #####################################
83-
# Configuration for the LLM model
86+
# Configuration for LLM models (group name starts with 'llm')
87+
# use 'llm' for the default LLM config
8488
##############################################################################
8589
[llm]
8690
# AWS access key ID
@@ -149,8 +153,18 @@ model = "gpt-4o"
149153
# Top p for the API
150154
#top_p = 0.5
151155

156+
[llm.gpt3]
157+
# API key to use
158+
api_key = "your-api-key"
159+
160+
# Model to use
161+
model = "gpt-3.5"
162+
152163
#################################### Agent ###################################
153-
# Configuration for the agent
164+
# Configuration for agents (group name starts with 'agent')
165+
# Use 'agent' for the default agent config
166+
# otherwise, group name must be `agent.<agent_name>` (case-sensitive), e.g.
167+
# agent.CodeActAgent
154168
##############################################################################
155169
[agent]
156170
# Memory enabled
@@ -159,8 +173,13 @@ model = "gpt-4o"
159173
# Memory maximum threads
160174
#memory_max_threads = 2
161175

162-
# Name of the agent
163-
#name = "CodeActAgent"
176+
# LLM config group to use
177+
#llm_config = 'llm'
178+
179+
[agent.RepoExplorerAgent]
180+
# Example: use a cheaper model for RepoExplorerAgent to reduce cost, especially
181+
# useful when an agent doesn't demand high quality but uses a lot of tokens
182+
llm_config = 'gpt3'
164183

165184
#################################### Sandbox ###################################
166185
# Configuration for the sandbox

‎docs/modules/usage/changelog.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
---
2+
sidebar_position: 8
3+
---
4+
5+
# Changelog
6+
7+
## 0.8 (release date: ??)
8+
9+
### Config breaking changes
10+
11+
In this release we introduced a few breaking changes to backend configurations.
12+
If you have only been using OpenDevin via frontend (web GUI), nothing needs
13+
to be taken care of.
14+
15+
Here's a list of breaking changes in configs. They only apply to users who
16+
use OpenDevin CLI via `main.py`. For more detail, see [#2756](https://github.com/OpenDevin/OpenDevin/pull/2756).
17+
18+
#### Removal of --model-name option from main.py
19+
20+
Please note that `--model-name`, or `-m` option, no longer exists. You should set up the LLM
21+
configs in `config.toml` or via environmental variables.
22+
23+
#### LLM config groups must be subgroups of 'llm'
24+
25+
Prior to release 0.8, you can use arbitrary name for llm config in `config.toml`, e.g.
26+
27+
```toml
28+
[gpt-4o]
29+
model="gpt-4o"
30+
api_key="<your_api_key>"
31+
```
32+
33+
and then use `--llm-config` CLI argument to specify the desired LLM config group
34+
by name. This no longer works. Instead, the config group must be under `llm` group,
35+
e.g.:
36+
37+
```toml
38+
[llm.gpt-4o]
39+
model="gpt-4o"
40+
api_key="<your_api_key>"
41+
```
42+
43+
If you have a config group named `llm`, no need to change it, it will be used
44+
as the default LLM config group.
45+
46+
#### 'agent' group no longer contains 'name' field
47+
48+
Prior to release 0.8, you may or may not have a config group named `agent` that
49+
looks like this:
50+
51+
```toml
52+
[agent]
53+
name="CodeActAgent"
54+
memory_max_threads=2
55+
```
56+
57+
Note the `name` field is now removed. Instead, you should put `default_agent` field
58+
under `core` group, e.g.
59+
60+
```toml
61+
[core]
62+
# other configs
63+
default_agent='CodeActAgent'
64+
65+
[agent]
66+
llm_config='llm'
67+
memory_max_threads=2
68+
69+
[agent.CodeActAgent]
70+
llm_config='gpt-4o'
71+
```
72+
73+
Note that similar to `llm` subgroups, you can also define `agent` subgroups.
74+
Moreover, an agent can be associated with a specific LLM config group. For more
75+
detail, see the examples in `config.template.toml`.

‎evaluation/TUTORIAL.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -53,14 +53,14 @@ api_key = "sk-XXX"
5353

5454
In this section, for the purpose of building an evaluation task, we don't use the standard OpenDevin web-based GUI, but rather run OpenDevin backend from CLI.
5555

56-
For example, you can run the following, which performs the specified task `-t`, with a particular model `-m` and agent `-c`, for a maximum number of iterations `-i`:
56+
For example, you can run the following, which performs the specified task `-t`, with a particular model config `-l` and agent `-c`, for a maximum number of iterations `-i`:
5757

5858
```bash
5959
poetry run python ./opendevin/core/main.py \
6060
-i 10 \
6161
-t "Write me a bash script that print hello world." \
6262
-c CodeActAgent \
63-
-m gpt-4o-2024-05-13
63+
-l llm
6464
```
6565

6666
After running the script, you will observe the following:

‎evaluation/agent_bench/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,12 +29,12 @@ enable_auto_lint = true
2929
box_type = "ssh"
3030
timeout = 120
3131

32-
[eval_gpt35_turbo]
32+
[llm.eval_gpt35_turbo]
3333
model = "gpt-3.5-turbo"
3434
api_key = "sk-123"
3535
temperature = 0.0
3636

37-
[eval_gpt4o]
37+
[llm.eval_gpt4o]
3838
model = "gpt-4o"
3939
api_key = "sk-123"
4040
temperature = 0.0

0 commit comments

Comments
 (0)