Skip to content

Commit 405c8a0

Browse files
xingyaowwtobitege
xingyaoww
and
tobitege
authored
[Arch] Add runtime image build CI & clean up runtime build using jinja2 template (OpenHands#3055)
* test_runtime_client.py to test _execute_bash() * runtime_build and runtime tweaks * fix in docker script * revert bash changes * use sandbox_config.update_source_code to control source code update * add od_version to the sandbox tag * add doc instruction for update source code * do not remove whole poetry folder; add mamba clean * add missing newlines * cleanup runtime dockerfile into jinja template * make prep temp file a separate function; make that function accessible through cli * modify `runtime_build.py` so it can generate directory for building docker img * add dockerfile and sdist of runtime to gitignore since it will be dynamically generated * add runtime to build * do not rebuild new image when an `od_runtime` is provided * use default container_image for testing if possible * move runtime tests to ghcr runtime workflow * update docker base dir for runtime * fix unittest * fix image name * fix image name for test case * rename to make it consistent --------- Co-authored-by: tobitege <[email protected]>
1 parent 547c510 commit 405c8a0

10 files changed

Lines changed: 441 additions & 166 deletions

File tree

‎.github/workflows/ghcr-runtime.yml‎

Lines changed: 242 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,242 @@
1+
name: Build Publish and Test Runtime Image
2+
3+
concurrency:
4+
group: ${{ github.workflow }}-${{ github.ref }}
5+
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
6+
7+
on:
8+
push:
9+
branches:
10+
- main
11+
tags:
12+
- '*'
13+
pull_request:
14+
workflow_dispatch:
15+
inputs:
16+
reason:
17+
description: 'Reason for manual trigger'
18+
required: true
19+
default: ''
20+
21+
jobs:
22+
ghcr_build_runtime:
23+
runs-on: ubuntu-latest
24+
25+
outputs:
26+
tags: ${{ steps.capture-tags.outputs.tags }}
27+
28+
permissions:
29+
contents: read
30+
packages: write
31+
32+
strategy:
33+
matrix:
34+
image: ["od_runtime"]
35+
base_image: ["ubuntu:22.04"]
36+
platform: ["amd64", "arm64"]
37+
38+
steps:
39+
- name: Checkout
40+
uses: actions/checkout@v4
41+
42+
- name: Free Disk Space (Ubuntu)
43+
uses: jlumbroso/free-disk-space@main
44+
with:
45+
# this might remove tools that are actually needed,
46+
# if set to "true" but frees about 6 GB
47+
tool-cache: true
48+
# all of these default to true, but feel free to set to
49+
# "false" if necessary for your workflow
50+
android: true
51+
dotnet: true
52+
haskell: true
53+
large-packages: true
54+
docker-images: false
55+
swap-storage: true
56+
57+
- name: Set up QEMU
58+
uses: docker/setup-qemu-action@v3
59+
60+
- name: Set up Docker Buildx
61+
id: buildx
62+
uses: docker/setup-buildx-action@v3
63+
64+
- name: Install poetry via pipx
65+
run: pipx install poetry
66+
67+
- name: Set up Python
68+
uses: actions/setup-python@v5
69+
with:
70+
python-version: "3.11"
71+
cache: "poetry"
72+
73+
- name: Install Python dependencies using Poetry
74+
run: make install-python-dependencies
75+
76+
- name: Create source distribution and Dockerfile
77+
run: poetry run python3 opendevin/runtime/utils/runtime_build.py --base_image ${{ matrix.base_image }} --build_folder containers/runtime
78+
79+
- name: Build and export image
80+
id: build
81+
run: ./containers/build.sh ${{ matrix.image }} ${{ github.repository_owner }} ${{ matrix.platform }}
82+
83+
- name: Capture tags
84+
id: capture-tags
85+
run: |
86+
tags=$(cat tags.txt)
87+
echo "tags=$tags"
88+
echo "tags=$tags" >> $GITHUB_OUTPUT
89+
90+
- name: Upload Docker image as artifact
91+
uses: actions/upload-artifact@v4
92+
with:
93+
name: ${{ matrix.image }}-docker-image-${{ matrix.platform }}
94+
path: /tmp/${{ matrix.image }}_image_${{ matrix.platform }}.tar
95+
96+
test-for-runtime:
97+
name: Test for Runtime
98+
runs-on: ubuntu-latest
99+
needs: ghcr_build_runtime
100+
env:
101+
PERSIST_SANDBOX: "false"
102+
steps:
103+
- uses: actions/checkout@v4
104+
105+
- name: Free Disk Space (Ubuntu)
106+
uses: jlumbroso/free-disk-space@main
107+
with:
108+
# this might remove tools that are actually needed,
109+
# when set to "true" but frees about 6 GB
110+
tool-cache: true
111+
112+
# all of these default to true, but feel free to set to
113+
# "false" if necessary for your workflow
114+
android: true
115+
dotnet: true
116+
haskell: true
117+
large-packages: true
118+
swap-storage: true
119+
120+
- name: Install poetry via pipx
121+
run: pipx install poetry
122+
123+
- name: Set up Python
124+
uses: actions/setup-python@v5
125+
with:
126+
python-version: "3.11"
127+
cache: "poetry"
128+
129+
- name: Install Python dependencies using Poetry
130+
run: make install-python-dependencies
131+
132+
- name: Download Runtime Docker image
133+
uses: actions/download-artifact@v4
134+
with:
135+
name: od_runtime-docker-image-amd64
136+
path: /tmp/
137+
138+
- name: Load Runtime image and run runtime tests
139+
run: |
140+
# Load the Docker image and capture the output
141+
output=$(docker load -i /tmp/od_runtime_image_amd64.tar)
142+
143+
# Extract the first image name from the output
144+
image_name=$(echo "$output" | grep -oP 'Loaded image: \K.*' | head -n 1)
145+
146+
# Print the full name of the image
147+
echo "Loaded Docker image: $image_name"
148+
149+
SANDBOX_CONTAINER_IMAGE=$image_name TEST_IN_CI=true poetry run pytest --cov=agenthub --cov=opendevin --cov-report=xml -s ./tests/unit/test_runtime.py
150+
151+
- name: Upload coverage to Codecov
152+
uses: codecov/codecov-action@v4
153+
env:
154+
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
155+
156+
ghcr_push:
157+
runs-on: ubuntu-latest
158+
# don't push if runtime tests fail
159+
needs: [ghcr_build_runtime, test-for-runtime]
160+
if: github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/')
161+
162+
env:
163+
tags: ${{ needs.ghcr_build_runtime.outputs.tags }}
164+
165+
permissions:
166+
contents: read
167+
packages: write
168+
169+
strategy:
170+
matrix:
171+
image: ["od_runtime"]
172+
platform: ["amd64", "arm64"]
173+
174+
steps:
175+
- name: Checkout code
176+
uses: actions/checkout@v4
177+
178+
- name: Login to GHCR
179+
uses: docker/login-action@v2
180+
with:
181+
registry: ghcr.io
182+
username: ${{ github.repository_owner }}
183+
password: ${{ secrets.GITHUB_TOKEN }}
184+
185+
- name: Download Docker images
186+
uses: actions/download-artifact@v4
187+
with:
188+
name: ${{ matrix.image }}-docker-image-${{ matrix.platform }}
189+
path: /tmp/${{ matrix.platform }}
190+
191+
- name: Load images and push to registry
192+
run: |
193+
mv /tmp/${{ matrix.platform }}/${{ matrix.image }}_image_${{ matrix.platform }}.tar .
194+
loaded_image=$(docker load -i ${{ matrix.image }}_image_${{ matrix.platform }}.tar | grep "Loaded image:" | head -n 1 | awk '{print $3}')
195+
echo "loaded image = $loaded_image"
196+
tags=$(echo ${tags} | tr ' ' '\n')
197+
image_name=$(echo "ghcr.io/${{ github.repository_owner }}/${{ matrix.image }}" | tr '[:upper:]' '[:lower:]')
198+
echo "image name = $image_name"
199+
for tag in $tags; do
200+
echo "tag = $tag"
201+
docker tag $loaded_image $image_name:${tag}_${{ matrix.platform }}
202+
docker push $image_name:${tag}_${{ matrix.platform }}
203+
done
204+
205+
create_manifest:
206+
runs-on: ubuntu-latest
207+
needs: [ghcr_build_runtime, ghcr_push]
208+
if: github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/')
209+
210+
env:
211+
tags: ${{ needs.ghcr_build_runtime.outputs.tags }}
212+
213+
strategy:
214+
matrix:
215+
image: ["od_runtime"]
216+
217+
permissions:
218+
contents: read
219+
packages: write
220+
221+
steps:
222+
- name: Checkout code
223+
uses: actions/checkout@v4
224+
225+
- name: Login to GHCR
226+
uses: docker/login-action@v2
227+
with:
228+
registry: ghcr.io
229+
username: ${{ github.repository_owner }}
230+
password: ${{ secrets.GITHUB_TOKEN }}
231+
232+
- name: Create and push multi-platform manifest
233+
run: |
234+
image_name=$(echo "ghcr.io/${{ github.repository_owner }}/${{ matrix.image }}" | tr '[:upper:]' '[:lower:]')
235+
echo "image name = $image_name"
236+
tags=$(echo ${tags} | tr ' ' '\n')
237+
for tag in $tags; do
238+
echo 'tag = $tag'
239+
docker buildx imagetools create --tag $image_name:$tag \
240+
$image_name:${tag}_amd64 \
241+
$image_name:${tag}_arm64
242+
done

‎.github/workflows/run-runtime-tests.yml‎

Lines changed: 0 additions & 64 deletions
This file was deleted.

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -220,3 +220,7 @@ image_build_logs
220220
run_instance_logs
221221

222222
od_runtime_*.tar
223+
224+
# docker build
225+
containers/runtime/Dockerfile
226+
containers/runtime/project.tar.gz

‎containers/build.sh‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,11 +27,14 @@ echo "Tags: ${tags[@]}"
2727

2828
if [[ "$image_name" == "opendevin" ]]; then
2929
dir="./containers/app"
30+
elif [[ "$image_name" == "od_runtime" ]]; then
31+
dir="./containers/runtime"
3032
else
3133
dir="./containers/$image_name"
3234
fi
3335

34-
if [[ ! -f "$dir/Dockerfile" ]]; then
36+
if [[ (! -f "$dir/Dockerfile") && "$image_name" != "od_runtime" ]]; then
37+
# Allow runtime to be built without a Dockerfile
3538
echo "No Dockerfile found"
3639
exit 1
3740
fi
@@ -46,6 +49,11 @@ if [[ -n "$org_name" ]]; then
4649
DOCKER_ORG="$org_name"
4750
fi
4851

52+
# If $DOCKER_IMAGE_TAG is set, add it to the tags
53+
if [[ -n "$DOCKER_IMAGE_TAG" ]]; then
54+
tags+=("$DOCKER_IMAGE_TAG")
55+
fi
56+
4957
DOCKER_REPOSITORY="$DOCKER_REGISTRY/$DOCKER_ORG/$DOCKER_IMAGE"
5058
DOCKER_REPOSITORY=${DOCKER_REPOSITORY,,} # lowercase
5159
echo "Repo: $DOCKER_REPOSITORY"

‎containers/runtime/README.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Dynamic constructed Dockerfile
2+
3+
This folder builds runtime image (sandbox), which will use a `Dockerfile` that is dynamically generated depends on the `base_image` AND a [Python source distribution](https://docs.python.org/3.10/distutils/sourcedist.html) that's based on the current commit of `opendevin`.
4+
5+
The following command will generate Dockerfile for `ubuntu:22.04` and the source distribution `.tar` into `containers/runtime`.
6+
7+
```bash
8+
poetry run python3 opendevin/runtime/utils/runtime_build.py \
9+
--base_image ubuntu:22.04 \
10+
--build_folder containers/runtime
11+
```

‎containers/runtime/config.sh‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
DOCKER_REGISTRY=ghcr.io
2+
DOCKER_ORG=opendevin
3+
DOCKER_BASE_DIR="./containers/runtime"
4+
# These two variables will be appended by the runtime_build.py script
5+
# DOCKER_IMAGE=
6+
# DOCKER_IMAGE_TAG=
7+
DOCKER_IMAGE=od_runtime
8+
DOCKER_IMAGE_TAG=od_v0.8.1_image_ubuntu_tag_22.04

0 commit comments

Comments
 (0)