Add Dockerfile and client.py; deploy to EC2 on AWS via Github Actions by abrichr · Pull Request #52 · microsoft/OmniParser

abrichr · 2024-10-28T22:20:51Z

Summary

This PR implements functionality to automatically deploy OmniParser to EC2 on AWS via Docker, Github Actions, and boto3.

It adds a Dockerfile which calls a download.py script for downloading the model weights, a deploy.py which automates deployment to AWS EC2 via Github Actions, and a working example client.py that supports command line usage:

python3.10 -m venv venv && source venv/bin/activate && pip install -r deploy_requirements.txt
python deploy.py start
python client.py "http://<server_ip>:7861"
python deploy.py status
python deploy.py ssh
python deploy.py stop

And API usage:

>>> from deploy import Deploy
>>> from client import predict
>>>
>>> # Start the deployment process
>>> Deploy.start()
>>> instance_id, instance_ip = Deploy.configure_ec2_instance()
>>> assert instance_ip, "Failed to retrieve instance IP address."
>>> print(f"Instance deployed at IP: {instance_ip}")
>>>
>>> # Use the deployed API programmatically with `client.py`
>>> api_url = f"http://{instance_ip}:7861"
>>> result = predict(api_url, "~/Desktop/screenshot.png")
Loaded as API: http://<instance_ip>:7861/ ✔
>>> result_data = result["result_data"]
>>> output_image_path = result['output_image']
>>> parsed_content_list = result_data["parsed_content_list"]
>>> label_coordinates = result_data["label_coordinates"]
>>> len(label_coordinates), len(parsed_content_list)
(80, 80)

It updates the README to include the following section, and another for deploying:

🚀 Docker Quick Start

Prerequisites:

CUDA-enabled GPU
Recent version of Docker

# Build the image (requires CUDA)
sudo docker build -t omniparser .

# Run the image
sudo docker run -d -p 7861:7861 --gpus all --name omniparser-container omniparser

Example successful Github Actions:

Based on microsoft/SoM#19.

Usage Instructions

Fork https://github.com/OpenAdaptAI/OmniParser, then clone your fork to your local machine and check out the feat/deploy branch:

git clone https://github.com/<your-username>/OmniParser.git
cd OmniParser
git checkout feat/deploy  # not required once PR#52 is merged

Or, until this is merged, you can add a new remote to your existing fork:

git remote add openadapt https://github.com/OpenAdaptAI/OmniParser
git fetch openadapt
git checkout feat/deploy

Follow the instructions at the top of deploy.py.

Requires the following values:

AWS_ACCESS_KEY_ID=<your aws access key id (required)>
AWS_SECRET_ACCESS_KEY=<your aws secret access key (required)>
AWS_REGION=<your aws region (required)>
GITHUB_OWNER=<your github owner (required)>  # e.g. abrichr
GITHUB_REPO=<your github repo (required)>    # e.g. OmniParse
GITHUB_TOKEN=<your github token (required)>
PROJECT_NAME=<your project name (required)>  # for tagging AWS resources

Example output

Deployment:

(venv) abrichr@MacBook-Pro-5 OmniParser % python deploy.py start                                                                                                                                                                                                                                                                                                                                                                                                                                                    
2024-10-29 11:35:55.759 | INFO     | __main__:set_github_secret:205 - set secret_name='AWS_ACCESS_KEY_ID'                                                                                                                                                                                                                                                                                                                                                                                                           
2024-10-29 11:35:56.303 | INFO     | __main__:set_github_secret:205 - set secret_name='AWS_SECRET_ACCESS_KEY'                                                                                                                                                                                                                                                                                                                                                                                                       
2024-10-29 11:35:56.849 | INFO     | __main__:set_github_secret:205 - set secret_name='SSH_PRIVATE_KEY'                                                                                                                                                                                                                                                                                                                                                                                                             
2024-10-29 11:35:58.240 | INFO     | __main__:get_or_create_security_group_id:307 - Created security group 'omniparser-SecurityGroup' with ID: sg-046ff0f92bfcf9e6f                                                                                                                                                                                                                                                                                                                                                 
2024-10-29 11:35:58.550 | INFO     | __main__:get_or_create_security_group_id:311 - Added inbound rules to allow access on ports=[22, 7861]                                                                                                                                                                                                                                                                                                                                                                         
2024-10-29 11:36:15.800 | INFO     | __main__:deploy_ec2_instance:414 - New instance created: ID - i-04075f30de683eee7, IP - 44.198.58.162                                                                                                                                                                                                                                                                                                                                                                          
2024-10-29 11:37:30.893 | ERROR    | __main__:configure_ec2_instance:456 - SSH connection attempt 1 failed: [Errno 60] Operation timed out                                                                                                                                                                                                                                                                                                                                                                          
2024-10-29 11:37:30.894 | INFO     | __main__:configure_ec2_instance:458 - Retrying SSH connection in 10 seconds...                                                                                                                                                                                                                                                                                                                                                                                                 
2024-10-29 11:37:41.516 | INFO     | __main__:configure_ec2_instance:482 - Executing command: sudo apt-get update                                                                                                                                                                                                                                                                                                                                                                                                   
2024-10-29 11:38:36.778 | INFO     | __main__:configure_ec2_instance:489 - Command executed successfully                                                                                                                                                                                                                                                                                                                                                                                                            
2024-10-29 11:38:36.779 | INFO     | __main__:configure_ec2_instance:482 - Executing command: sudo apt-get install -y docker.io                                                                                                                                                                                                                                                                                                                                                                                     
2024-10-29 11:39:58.567 | INFO     | __main__:configure_ec2_instance:489 - Command executed successfully                                                                                                                                                                                                                                                                                                                                                                                                            
2024-10-29 11:39:58.568 | INFO     | __main__:configure_ec2_instance:482 - Executing command: sudo systemctl start docker                                                                                                                                                                                                                                                                                                                                                                                           
2024-10-29 11:39:58.876 | INFO     | __main__:configure_ec2_instance:489 - Command executed successfully                                                                                                                                                                                                                                                                                                                                                                                                            
2024-10-29 11:39:58.876 | INFO     | __main__:configure_ec2_instance:482 - Executing command: sudo systemctl enable docker                                                                                                                                                                                                                                                                                                                                                                                          
2024-10-29 11:39:59.636 | INFO     | __main__:configure_ec2_instance:489 - Command executed successfully                                                                                                                                                                                                                                                                                                                                                                                                            
2024-10-29 11:39:59.636 | INFO     | __main__:configure_ec2_instance:482 - Executing command: sudo usermod -a -G docker ${USER}                                                                                                                                                                                                                                                                                                                                                                                     
2024-10-29 11:39:59.881 | INFO     | __main__:configure_ec2_instance:489 - Command executed successfully                                                                                                                                                                                                                                                                                                                                                                                                            
2024-10-29 11:39:59.881 | INFO     | __main__:configure_ec2_instance:482 - Executing command: sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose                                                                                                                                                                                                                                                                     ]
2024-10-29 11:40:00.235 | INFO     | __main__:configure_ec2_instance:489 - Command executed successfully                                                                                                                                                                                                                                                                                                                                                                                                             
2024-10-29 11:40:00.235 | INFO     | __main__:configure_ec2_instance:482 - Executing command: sudo chmod +x /usr/local/bin/docker-compose 
2024-10-29 11:40:00.405 | INFO     | __main__:configure_ec2_instance:489 - Command executed successfully                                                                                                                                                   
2024-10-29 11:40:00.406 | INFO     | __main__:configure_ec2_instance:482 - Executing command: sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose 
2024-10-29 11:40:00.560 | INFO     | __main__:configure_ec2_instance:489 - Command executed successfully                                                                                                                                                   
2024-10-29 11:40:00.560 | INFO     | __main__:configure_ec2_instance:482 - Executing command: sudo docker system prune -af --volumes 
2024-10-29 11:40:00.886 | INFO     | __main__:configure_ec2_instance:489 - Command executed successfully                                                                                                                                                   
2024-10-29 11:40:01.795 | INFO     | __main__:get_or_create_security_group_id:276 - Security group 'omniparser-SecurityGroup' already exists: sg-046ff0f92bfcf9e6f 
2024-10-29 11:40:02.030 | INFO     | __main__:get_or_create_security_group_id:287 - Rule for port 22 already exists                                                                                                                                        
2024-10-29 11:40:02.462 | INFO     | __main__:get_or_create_security_group_id:287 - Rule for port 7861 already exists         
2024-10-29 11:40:02.824 | INFO     | __main__:deploy_ec2_instance:366 - Instance already running: ID - i-04075f30de683eee7, IP - 44.198.58.162 
2024-10-29 11:40:02.838 | INFO     | __main__:generate_github_actions_workflow:535 - GitHub Actions EC2 workflow file generated successfully.                                                                                                              
2024-10-29 11:40:02.928 | INFO     | __main__:update_git_remote_with_pat:611 - Git remote 'origin' updated successfully.                                                                                                                                   
diff --git a/.github/workflows/docker-build-ec2.yml b/.github/workflows/docker-build-ec2.yml                                  
new file mode 100644                                                                                                         
index 0000000..4669196                                                                                                       
--- /dev/null                                                                                                                
+++ b/.github/workflows/docker-build-ec2.yml                                                                                 
@@ -0,0 +1,41 @@                                                                                                             
+# Autogenerated via deploy.py, do not edit!                                                                                 
+                                                                                                                            
+name: Docker Build on EC2 Instance for OmniParser                                                                           
+                                                                                                                            
+on:                                                                                                                         
+  push:                                                                                                                     
+    branches:                                                                                                               
+      - feat/deploy                                                                                                         
+                                                                                                                            
+jobs:                                                                                                                       
+  build:                                                                                                                    
+    runs-on: ubuntu-latest                                                                                                  
+    steps:                                                                                                                  
+      - name: Checkout code                                                                                                 
+        uses: actions/checkout@v2                                                                                           
+                                                                                                                            
+      - name: SSH and Execute Build on EC2                                                                                  
+        uses: appleboy/ssh-action@master                                                                                    
+        with:                                                                                                               
+          command_timeout: "60m"                                                                                            
+          host: 44.198.58.162                                                                                               
+          username: ubuntu                                                                                                  
+                                                                                                                            
+          key: ${{ secrets.SSH_PRIVATE_KEY }}                                                                               
+                                                                                                                            
+          script: |                                                                                                         
+            rm -rf OmniParser || true                                                                                       
+            git clone https://github.com/OpenAdaptAI/OmniParser                                                             
+            cd OmniParser                                                                                                   
+            git checkout feat/deploy                                                                                        
+            git pull                                                                                                        
+                                                                                                                            
+            # Stop and remove any existing containers                                                                       
+            sudo docker stop omniparser-container || true                                                                   
+            sudo docker rm omniparser-container || true                                                                     
+                                                                                                                            
+            # Build the Docker image                                                                                        
+            sudo nvidia-docker build -t omniparser .                                                                        
+                                                                                                                            
+            # Run the Docker container on the specified port                                                                
+            sudo docker run -d -p 7861:7861 --gpus all --name omniparser-container omniparser                               
[feat/deploy 7b8d20c] add workflow file                                                                                      
 1 file changed, 41 insertions(+)                                                                                            
 create mode 100644 .github/workflows/docker-build-ec2.yml                                                                                                                                                                                                
2024-10-29 11:40:03.054 | INFO     | __main__:start:645 - Changes committed successfully.                                    
warning: current Git remote contains credentials                                                                                                                                                                                                          
Enumerating objects: 6, done.                                                                                                
Counting objects: 100% (6/6), done.                                                                                          
Delta compression using up to 12 threads                                                                                     
Compressing objects: 100% (3/3), done.                                                                                       
Writing objects: 100% (5/5), 891 bytes | 891.00 KiB/s, done.                                                                 
Total 5 (delta 1), reused 0 (delta 0), pack-reused 0                                                                         
remote: Resolving deltas: 100% (1/1), completed with 1 local object.                                                         
To https://github.com/OpenAdaptAI/OmniParser.git                                                                             
   0657932..7b8d20c  feat/deploy -> feat/deploy                                                                              
branch 'feat/deploy' set up to track 'origin/feat/deploy'.                                                                   
2024-10-29 11:40:04.649 | INFO     | __main__:git_push_set_upstream:582 - Branch 'feat/deploy' pushed and set up to track 'origin/feat/deploy'.
2024-10-29 11:40:04.651 | INFO     | __main__:start:660 - Deployment process completed.                                                                                                                                                                   
2024-10-29 11:40:04.651 | INFO     | __main__:start:661 - Check the GitHub Actions at https://github.com/OpenAdaptAI/OmniParser/actions.
2024-10-29 11:40:04.651 | INFO     | __main__:start:662 - Once the action is complete, run:                                                                                                                                                               
2024-10-29 11:40:04.651 | INFO     | __main__:start:663 -     python client.py http://44.198.58.162:7861                     
(venv) abrichr@MacBook-Pro-5 OmniParser % git status

Redeployment:

(venv) abrichr@MacBook-Pro-5 OmniParser % python deploy.py start
2024-10-29 11:16:39.718 | INFO     | __main__:set_github_secret:205 - set secret_name='AWS_ACCESS_KEY_ID'
2024-10-29 11:16:40.346 | INFO     | __main__:set_github_secret:205 - set secret_name='AWS_SECRET_ACCESS_KEY'
2024-10-29 11:16:41.268 | INFO     | __main__:set_github_secret:205 - set secret_name='SSH_PRIVATE_KEY'
2024-10-29 11:16:41.952 | INFO     | __main__:get_or_create_security_group_id:276 - Security group 'omniparser-SecurityGroup' already exists: sg-0f89ae01d0689defd
2024-10-29 11:16:42.158 | INFO     | __main__:get_or_create_security_group_id:287 - Rule for port 22 already exists
2024-10-29 11:16:42.512 | INFO     | __main__:get_or_create_security_group_id:287 - Rule for port 7861 already exists
2024-10-29 11:16:42.895 | INFO     | __main__:deploy_ec2_instance:366 - Instance already running: ID - i-09c48e242dba6fa58, IP - 34.206.53.77
2024-10-29 11:16:43.429 | INFO     | __main__:configure_ec2_instance:482 - Executing command: sudo apt-get update
2024-10-29 11:16:45.848 | INFO     | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:16:45.848 | INFO     | __main__:configure_ec2_instance:482 - Executing command: sudo apt-get install -y docker.io
2024-10-29 11:16:46.476 | INFO     | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:16:46.476 | INFO     | __main__:configure_ec2_instance:482 - Executing command: sudo systemctl start docker
2024-10-29 11:16:46.625 | INFO     | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:16:46.626 | INFO     | __main__:configure_ec2_instance:482 - Executing command: sudo systemctl enable docker
2024-10-29 11:16:47.557 | INFO     | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:16:47.558 | INFO     | __main__:configure_ec2_instance:482 - Executing command: sudo usermod -a -G docker ${USER}
2024-10-29 11:16:47.871 | INFO     | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:16:47.872 | INFO     | __main__:configure_ec2_instance:482 - Executing command: sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
2024-10-29 11:16:48.156 | INFO     | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:16:48.156 | INFO     | __main__:configure_ec2_instance:482 - Executing command: sudo chmod +x /usr/local/bin/docker-compose
2024-10-29 11:16:48.432 | INFO     | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:16:48.433 | INFO     | __main__:configure_ec2_instance:482 - Executing command: sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
2024-10-29 11:16:48.624 | ERROR    | __main__:configure_ec2_instance:498 - Error in command: sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose, Exit Status: 1, Error: b"ln: failed to create symbolic link '/usr/bin/docker-compose': File exists\n"
2024-10-29 11:16:48.625 | INFO     | __main__:configure_ec2_instance:482 - Executing command: sudo docker system prune -af --volumes
2024-10-29 11:16:55.334 | INFO     | __main__:configure_ec2_instance:489 - Command executed successfully
2024-10-29 11:16:56.012 | INFO     | __main__:get_or_create_security_group_id:276 - Security group 'omniparser-SecurityGroup' already exists: sg-0f89ae01d0689defd
2024-10-29 11:16:56.272 | INFO     | __main__:get_or_create_security_group_id:287 - Rule for port 22 already exists
2024-10-29 11:16:56.622 | INFO     | __main__:get_or_create_security_group_id:287 - Rule for port 7861 already exists
2024-10-29 11:16:56.959 | INFO     | __main__:deploy_ec2_instance:366 - Instance already running: ID - i-09c48e242dba6fa58, IP - 34.206.53.77
2024-10-29 11:16:56.968 | INFO     | __main__:generate_github_actions_workflow:535 - GitHub Actions EC2 workflow file generated successfully.
2024-10-29 11:16:57.015 | INFO     | __main__:update_git_remote_with_pat:611 - Git remote 'origin' updated successfully.
2024-10-29 11:16:57.044 | INFO     | __main__:start:647 - No changes to commit.
branch 'feat/deploy' set up to track 'origin/feat/deploy'.
Everything up-to-date
2024-10-29 11:16:57.968 | INFO     | __main__:git_push_set_upstream:582 - Branch 'feat/deploy' pushed and set up to track 'origin/feat/deploy'.
2024-10-29 11:16:57.969 | INFO     | __main__:start:660 - Deployment process completed.
2024-10-29 11:16:57.969 | INFO     | __main__:start:661 - Check the GitHub Actions at https://github.com/OpenAdaptAI/OmniParser/actions.
2024-10-29 11:16:57.969 | INFO     | __main__:start:662 - Once the action is complete, run:
2024-10-29 11:16:57.969 | INFO     | __main__:start:663 -     python client.py http://34.206.53.77:7861 <path>/<to>/<image.png>

Command line client:

(venv) abrichr@MacBook-Pro-5 OmniParser % python client.py http://44.198.58.162:7861 ~/oa/OpenAdapt/tests/assets/excel.png                                                                                                                                
Loaded as API: http://44.198.58.162:7861/ ✔                                                                                                                                                                                                               
2024-10-29 18:55:51.336 | INFO     | __main__:predict:61 - Prediction completed successfully                                                                                                                                                              
2024-10-29 18:55:51.337 | INFO     | __main__:predict:70 - label_coordinates={'0': [0.12276943773031235, 0.002296212362125516, 0.045681655406951904, 0.028702640905976295], '1': [0.48429691791534424, 0.008610793389379978, 0.030335474759340286, 0.01607
3478385806084], '2': [0.0060670948587358, 0.043628014624118805, 0.02890792302787304, 0.016073478385806084], '3': [0.049250528216362, 0.043628014624118805, 0.02855103462934494, 0.016073478385806084], '4': [0.0935046374797821, 0.043628014624118805, 0.0
242683794349432, 0.016073478385806084], '5': [0.13276232779026031, 0.04247990995645523, 0.057102058082818985, 0.02238805964589119], '6': [0.20628122985363007, 0.043628014624118805, 0.04282655194401741, 0.016073478385806084], '7': [0.26552459597587585
, 0.043628014624118805, 0.021770164370536804, 0.016073478385806084], '8': [0.3029978573322296, 0.04247990995645523, 0.03283369168639183, 0.017221584916114807], '9': [0.3508208394050598, 0.043628014624118805, 0.02319771610200405, 0.016073478385806084]
, '10': [0.39971449971199036, 0.043628014624118805, 0.03354746475815773, 0.016073478385806084], '11': [0.9625267386436462, 0.04649827629327774, 0.026409707963466644, 0.016073478385806084], '12': [0.08422555029392242, 0.08553387224674225, 0.0578158460
5574608, 0.01836968958377838], '13': [0.4057815968990326, 0.08610792458057404, 0.03854389861226082, 0.017221584916114807], '14': [0.4921484589576721, 0.08553387224674225, 0.03354746475815773, 0.016073478385806084], '15': [0.012491078115999699, 0.1194
0297484397888, 0.021770164370536804, 0.014925390481948853], '16': [0.08529621362686157, 0.11997702717781067, 0.06495360285043716, 0.02238805964589119], '17': [0.40613847970962524, 0.1228473111987114, 0.0239114910364151, 0.016073478385806084], '18': [
0.4382583796977997, 0.1228473111987114, 0.026409707963466644, 0.014925372786819935], '19': [0.6145610213279724, 0.11710677295923233, 0.04318344220519066, 0.017221584916114807], '20': [0.6623840928077698, 0.11940297484397888, 0.027837172150611877, 0.0
14925390481948853], '21': [0.7041398882865906, 0.11997703462839127, 0.015703069046139717, 0.01205510925501585], '22': [0.7494646906852722, 0.11997703462839127, 0.022483939304947853, 0.01205510925501585], '23': [0.7847965955734253, 0.11997703462839127
, 0.0239114910364151, 0.01205510925501585], '24': [0.8183440566062927, 0.1188289225101471, 0.02855103462934494, 0.015499443747103214], '25': [0.8954318165779114, 0.11997703462839127, 0.017130620777606964, 0.01205510925501585], '26': [0.93076366186141
97, 0.11997703462839127, 0.016416845843195915, 0.01205510925501585], '27': [0.6145610213279724, 0.13318024575710297, 0.04139900207519531, 0.016073478385806084], '28': [0.6602426767349243, 0.13375429809093475, 0.031406138092279434, 0.01205510925501585
], '29': [0.699857234954834, 0.13318024575710297, 0.0242683794349432, 0.014925372786819935], '30': [0.8979300260543823, 0.13375429809093475, 0.01927194930613041, 0.01205510925501585], '31': [0.9304068684577942, 0.13318024575710297, 0.0249821562319993
97, 0.014925372786819935], '32': [0.02105638198554516, 0.16590125858783722, 0.059957172721624374, 0.01836968958377838], '33': [0.08422557264566422, 0.16532720625400543, 0.3875803053379059, 0.01894374191761017], '34': [0.9107779264450073, 0.1659012585
8783722, 0.07708779722452164, 0.017221584916114807], '35': [0.050321198999881744, 0.19517795741558075, 0.07566024363040924, 0.024684270843863487], '36': [0.18022841215133667, 0.30022960901260376, 0.10706637054681778, 0.051090698689222336], '37': [0.3
1334760785102844, 0.3053961396217346, 0.12312633544206619, 0.04477611929178238], '38': [0.4464668035507202, 0.3071182668209076, 0.055317629128694534, 0.03731343150138855], '39': [0.581013560295105, 0.305970162153244, 0.06745181977748871, 0.0373134315
0138855], '40': [0.014989293180406094, 0.3639494776725769, 0.013918629847466946, 0.029276693239808083], '41': [0.09671662747859955, 0.3559127151966095, 0.08065667748451233, 0.0482204370200634], '42': [0.29407569766044617, 0.3639494776725769, 0.013918
586075305939, 0.029276693239808083], '43': [0.4275517165660858, 0.3639494776725769, 0.013204853981733322, 0.029276693239808083], '44': [0.014989293180406094, 0.4161882996559143, 0.013918629847466946, 0.02985074557363987], '45': [0.09743039309978485, 
0.4087255895137787, 0.07994289696216583, 0.0482204370200634], '46': [0.29407569766044617, 0.4161882996559143, 0.013918586075305939, 0.02985074557363987], '47': [0.4275517463684082, 0.4161882996559143, 0.013918629847466946, 0.02985074557363987], '48':
 [0.0970735102891922, 0.4609643816947937, 0.08065667748451233, 0.04879448935389519], '49': [0.29407569766044617, 0.46785303950309753, 0.013918586075305939, 0.030998852103948593], '50': [0.4275517463684082, 0.4690011441707611, 0.013918629847466946, 0.
02985074557363987], '51': [0.5613847374916077, 0.4690011441707611, 0.013561741448938847, 0.02985074557363987], '52': [0.6948608160018921, 0.46785303950309753, 0.013918629847466946, 0.030998852103948593], '53': [0.09671661257743835, 0.5143513679504395
, 0.08137047290802002, 0.0482204370200634], '54': [0.29407569766044617, 0.5206659436225891, 0.013918586075305939, 0.030998852103948593], '55': [0.4275517463684082, 0.5206659436225891, 0.013918629847466946, 0.030998852103948593], '56': [0.561384737491
6077, 0.5206659436225891, 0.013561741448938847, 0.030998852103948593], '57': [0.6948608160018921, 0.5206659436225891, 0.013918629847466946, 0.030998852103948593], '58': [0.008922198787331581, 0.7824339866638184, 0.02783725969493389, 0.036739379167556
76], '59': [0.008208423852920532, 0.8363949656486511, 0.027123482897877693, 0.034443169832229614], '60': [0.008922199718654156, 0.8892077803611755, 0.027123482897877693, 0.034443169832229614], '61': [0.055317629128694534, 0.9500574469566345, 0.029621
697962284088, 0.014925372786819935], '62': [0.015346181578934193, 0.9787600636482239, 0.027480371296405792, 0.017221584916114807], '63': [0.06780871003866196, 0.9787600636482239, 0.10135617107152939, 0.017221584916114807], '64': [0.9586009979248047, 
0.9787600636482239, 0.024625267833471298, 0.014925372786819935], '65': [0.6651304364204407, 0.08138538151979446, 0.022024760022759438, 0.03468729183077812], '66': [0.7468552589416504, 0.07959043234586716, 0.023525850847363472, 0.028781019151210785], 
'67': [0.6991221308708191, 0.08180632442235947, 0.018437061458826065, 0.03362924978137016], '68': [0.7826605439186096, 0.0806342288851738, 0.02190661057829857, 0.027261780574917793], '69': [0.6214930415153503, 0.08046738058328629, 0.01830035261809826
, 0.031031927093863487], '70': [0.8183373212814331, 0.07944951951503754, 0.021337995305657387, 0.03104429505765438], '71': [0.09984152764081955, 0.9439650177955627, 0.02320893481373787, 0.028721420094370842], '72': [0.38827988505363464, 0.11905978620
052338, 0.01550249382853508, 0.02412971667945385], '73': [0.866210401058197, 0.08966667205095291, 0.015609142370522022, 0.03238759562373161], '74': [0.3112151622772217, 0.07979363203048706, 0.01715545356273651, 0.030124405398964882], '75': [0.0003453
716926742345, 0.9426923394203186, 0.044026125222444534, 0.029541436582803726], '76': [0.6377907395362854, 0.2446022927761078, 0.015758832916617393, 0.03145069256424904], '77': [0.3110503554344177, 0.12188976258039474, 0.017567103728652, 0.02876365743
5774803], '78': [0.8960394263267517, 0.08187582343816757, 0.020451003685593605, 0.02726871706545353], '79': [0.9309751987457275, 0.0796903744339943, 0.02361646667122841, 0.029221685603260994]}                                                          
2024-10-29 18:55:51.339 | INFO     | __main__:predict:71 - parsed_content_list=['Text Box ID 0: 5 v G', 'Text Box ID 1: Book1', 'Text Box ID 2: Home', 'Text Box ID 3: Insert', 'Text Box ID 4: Draw', 'Text Box ID 5: Page Layout', 'Text Box ID 6: Formu
las', 'Text Box ID 7: Data', 'Text Box ID 8: Review', 'Text Box ID 9: View', 'Text Box ID 10: Tell me', 'Text Box ID 11: Share', 'Text Box ID 12: Calibri (Body)', 'Text Box ID 13: Wrap Text', 'Text Box ID 14: General', 'Text Box ID 15: Paste', 'Text 
Box ID 16: B   I W ~ [', 'Text Box ID 17: Merge', 'Text Box ID 18: Center', 'Text Box ID 19: Conditional', 'Text Box ID 20: Format', 'Text Box ID 21: Cell', 'Text Box ID 22: Insert', 'Text Box ID 23: Delete', 'Text Box ID 24: Format', 'Text Box ID 25
: Sort', 'Text Box ID 26: Find', 'Text Box ID 27: Formatting', 'Text Box ID 28: as Table', 'Text Box ID 29: Styles', 'Text Box ID 30: Filter', 'Text Box ID 31: Select', 'Text Box ID 32: Office Update', 'Text Box ID 33: To keep up-to-date with securit
y updates, fixes, and improvements, choose Check for Updates:', 'Text Box ID 34: Check for Updates', 'Text Box ID 35: 4* v fx', 'Text Box ID 36: Marketing', 'Text Box ID 37: Engineering', 'Text Box ID 38: Sales', 'Text Box ID 39: Admin', 'Text Box ID
 40: 2', 'Text Box ID 41: 13-May', 'Text Box ID 42: 2', 'Text Box ID 43: 3', 'Text Box ID 44: 3', 'Text Box ID 45: 14-May', 'Text Box ID 46: 3', 'Text Box ID 47: 2', 'Text Box ID 48: 15-May', 'Text Box ID 49: 3', 'Text Box ID 50: 3', 'Text Box ID 51:
 3', 'Text Box ID 52: 3', 'Text Box ID 53: 16-May', 'Text Box ID 54: 2', 'Text Box ID 55: 2', 'Text Box ID 56: 2', 'Text Box ID 57: 1', 'Text Box ID 58: 10', 'Text Box ID 59: 11', 'Text Box ID 60: 12', 'Text Box ID 61: Sheet1', 'Text Box ID 62: Ready
', 'Text Box ID 63: Accessibility: Good to go', 'Text Box ID 64: 287%', 'Icon Box ID 65: a drawing or painting tool.', 'Icon Box ID 66: a layout or table layout.', 'Icon Box ID 67: a drawing or painting tool.', 'Icon Box ID 68: expanding or opening a
 window or view.', 'Icon Box ID 69: a table or spreadsheet.', 'Icon Box ID 70: a table or spreadsheet.', 'Icon Box ID 71: adding or creating a new item.', 'Icon Box ID 72: Decrease', 'Icon Box ID 73: Download', 'Icon Box ID 74: a menu or list.', 'Ico
n Box ID 75: navigating to the previous item or screen.', 'Icon Box ID 76: the "E" function.', 'Icon Box ID 77: Justified', 'Icon Box ID 78: Text formatting options.', 'Icon Box ID 79: a search function.']
2024-10-29 18:55:51.341 | INFO     | __main__:predict:77 - Parsed content saved to: result_data_20241029_185545.json         
2024-10-29 18:55:51.343 | INFO     | __main__:predict:83 - Output image saved to: output_image_20241029_185545.png  
(venv) abrichr@MacBook-Pro-5 OmniParser % open output_image.png 
(venv) abrichr@MacBook-Pro-5 OmniParser % head result_data_20241029_185545.json                                              
{                                                                                                                            
    "label_coordinates": {                                                                                                   
        "0": [                                                                                                               
            0.12276943773031235,                                                                                             
            0.002296212362125516,                                                                                            
            0.045681655406951904,                                                                                            
            0.028702640905976295                                                                                             
        ],                                                                                                                   
        "1": [                                                                                                               
            0.48429691791534424,                                                                                             
(venv) abrichr@MacBook-Pro-5 OmniParser % tail result_data_20241029_185545.json                                              
        "Icon Box ID 72: Decrease",                                                                                          
        "Icon Box ID 73: Download",                                                                                          
        "Icon Box ID 74: a menu or list.",                                                                                   
        "Icon Box ID 75: navigating to the previous item or screen.",                                                        
        "Icon Box ID 76: the \"E\" function.",                                                                               
        "Icon Box ID 77: Justified",                                                                                         
        "Icon Box ID 78: Text formatting options.",                                                                          
        "Icon Box ID 79: a search function."                                                                                 
    ]                                                                                                                        
}%

Client API:

>>> from client import predict
>>> from PIL import Image
>>> 
>>> result = predict("http://98.80.191.41:7861", "~/Desktop/screenshot.png")
Loaded as API: http://98.80.191.41:7861/ ✔
>>> result_data = result["result_data"]
>>> label_coordinates = result_data["label_coordinates"]
>>> parsed_content_list = result_data["parsed_content_list"]
>>> len(label_coordinates), len(parsed_content_list)
(80, 80)
>>> output_image_path = result['output_image']
>>> Image.open(output_image_path).show()

Status:

(venv) abrichr@MacBook-Pro-5 OmniParser % python deploy.py status
2024-10-29 18:04:25.726 | INFO     | __main__:status:748 - Instance ID: i-04075f30de683eee7, State: running, HTTP URL: http://44.198.58.162:7861

Termination:

(venv) abrichr@MacBook-Pro-5 OmniParser % python deploy.py stop
2024-10-29 11:25:03.330 | INFO     | __main__:stop:716 - Terminating instance: ID - i-09c48e242dba6fa58
2024-10-29 11:31:24.090 | INFO     | __main__:stop:719 - Instance i-09c48e242dba6fa58 terminated successfully.
2024-10-29 11:31:24.604 | INFO     | __main__:stop:724 - Deleted security group: omniparser-SecurityGroup
(venv) abrichr@MacBook-Pro-5 OmniParser %

(Note that all AWS configuration is automated.)

…-build-ec2.yml.j2; download.py; entrypoint.sh; requirements.txt

Deploy to EC2 on AWS via Github Actions

Restore requirements.txt

abrichr · 2024-10-29T23:24:10Z

@yadong-lu thank you for the excellent work on this repo!

I believe this PR is ready to merge. Please let me know if you have any questions/comments!

yadong-lu · 2024-11-01T03:08:12Z

Hi @abrichr, really appreciate this pr! since it contains a lot of changes, do you think we can keep the changes only for docker and aws deployment. E.g. can we remove the change for gradio_demo.py? Thanks.

abrichr · 2024-11-01T15:35:12Z

@yadong-lu I have reverted gradio_demo.py and added only the minimum required changes in order to return the bboxes (in order to implement a working client.py) and format the output in the text box with new lines (as was implemented previously):

% python client.py http://3.238.21.15:7861  ~/oa/OpenAdapt/tests/assets/excel.png
Loaded as API: http://3.238.21.15:7861/ ✔
2024-11-01 11:34:48.401 | INFO     | __main__:predict_and_save:116 - Parsed content saved to: result_data_20241101_113437.json
2024-11-01 11:34:48.404 | INFO     | __main__:predict_and_save:122 - Output image saved to: output_image_20241101_113437.png
(venv) abrichr@MacBook-Pro-5 OmniParser % 
(venv) abrichr@MacBook-Pro-5 OmniParser % head result_data_20241101_113437.json
{
    "parsed_content_list": [
        "Text Box ID 0: OO0",
        "Text Box ID 1: A   9C .",
        "Text Box ID 2: Book1",
        "Text Box ID 3: Q",
        "Text Box ID 4: Home",
        "Text Box ID 5:  Insert",
        "Text Box ID 6: Draw",
        "Text Box ID 7: Page Layout",
(venv) abrichr@MacBook-Pro-5 OmniParser % tail result_data_20241101_113437.json
            0.02726871740120013
        ],
        "95": [
            0.9309752067440668,
            0.0796903748189267,
            0.0236164667536581,
            0.02922168473288605
        ]
    }
}%

(Without returning label_coordinates in gradio_demo.py::process(), the client doesn't have access to the label_coordinates, which makes it non-functional.)

You can see the resulting successful Github action here: https://github.com/OpenAdaptAI/OmniParser/actions/runs/11631460372/job/32392578265

Ready for re-review.

Edit: if you prefer I can create a new gradio_demo_bboxes.py to use in deploy.py and client.py, and leave the existing gradio_demo.py untouched. Please let me know your preference 🙏

…a-docker with docker; improve documentation

This reverts commit 10b08e6.

n-sviridenko · 2025-01-21T09:08:21Z

@abrichr I'd also add this to download.py:

from utils import reader as easyocr_reader
print(f"Downloaded EasyOCR model")

from utils import paddle_ocr
print(f"Downloaded PaddleOCR model")

So that it doesn't spend GPUs on that

abrichr · 2025-02-18T21:58:46Z

See OpenAdaptAI/OpenAdapt#943 for an updated version that works independently of Github Actions and is compatible with OmniParser v2 (weights and FastAPI server).

@n-sviridenko thank you for the suggestion! Implemented here: OpenAdaptAI/OpenAdapt@6290d17

abrichr mentioned this pull request Oct 29, 2024

Bedrock API? suitedaces/computer-agent#2

Open

Add Dockerfile; client.py; deploy.py; deploy_requirements.txt; docker…

f8f21f6

…-build-ec2.yml.j2; download.py; entrypoint.sh; requirements.txt

abrichr force-pushed the feat/deploy branch from b53fc82 to f8f21f6 Compare October 29, 2024 15:23

abrichr changed the title ~~Add Dockerfile and client.py; deploy to EC2 on AWS via Github Actions (work in progress)~~ Add Dockerfile and client.py; deploy to EC2 on AWS via Github Actions Oct 29, 2024

abrichr added 7 commits October 29, 2024 11:35

documentation

0657932

add workflow file

7b8d20c

improve logging

71892c3

feat(deploy): Add Dockerfile, client.py, deploy.py

4fdd738

Deploy to EC2 on AWS via Github Actions

remove autogenerated .github/workflows/docker-build-ec2.yml

31c6289

restore requirements.txt

6c55805

add workflow file

f31f8ae

abrichr mentioned this pull request Oct 29, 2024

Where I can get 'weights/omniparser/icon_caption_blip2/best.pt' #7

Open

abrichr added 9 commits October 29, 2024 16:04

append timestamps to filenames; return bboxes

1e6fa93

add missing json import

b439503

return json with label_coordinates and parsed_content_list

be7866f

remove conflicting container if it exists

faa075e

NumpyEncoder

db9b1ee

save raw result data

706f521

improve documentation

a1b85c5

feat(client): save bounding boxes

803d0bf

Restore requirements.txt

Add functionality to save bounding boxes

88f7246

abrichr added 3 commits October 29, 2024 19:43

update README

754d0b7

improve README

af8c9da

add deploy section to README

4fdb813

abrichr mentioned this pull request Oct 30, 2024

struggling to learn how to use/interact with the functionality of the model #12

Open

improve documentation

3520928

This was referenced Oct 30, 2024

Use hf_hub_download in the gradio demo to take advantage of cache #11

Closed

No such file or directory: 'weights/icon_caption_florence/configuration_florence2.py' #54

Open

SafetensorError: Error while deserializing header: HeaderTooLarge #34

Open

undo changes to gradio_demo.py

b8b952c

abrichr marked this pull request as draft November 1, 2024 03:54

This was referenced Nov 1, 2024

本地部署教程 #66

Open

Florence2 VS BLIP2 #61

Closed

abrichr force-pushed the feat/deploy branch from a92eefc to b8b952c Compare November 1, 2024 15:18

abrichr marked this pull request as ready for review November 1, 2024 15:18

Add JSON output formatting to process function; return label_coordinates

201af0f

abrichr force-pushed the feat/deploy branch from c15ccd3 to 201af0f Compare November 1, 2024 15:33

abrichr mentioned this pull request Nov 1, 2024

Feat/deploy5 #67

Closed

abrichr added 5 commits November 1, 2024 14:12

Update Dockerfile documentation

a411848

Merge branch 'master' into feat/deploy

b706744

remove superfluous print

9ad451a

more terse

9f2dc91

parsed_content

76d6110

abrichr mentioned this pull request Nov 12, 2024

Dockerfile #82

Closed

abrichr added 5 commits November 12, 2024 15:02

simplify dependencies; update AMI; ssh non_interactive; replace nvidi…

16a4d7a

…a-docker with docker; improve documentation

add workflow file

10b08e6

Revert "add workflow file"

2b91690

This reverts commit 10b08e6.

update documentation

fcb23b8

Update README.md: replace nvidia-docker with docker

cd69e3a

abrichr mentioned this pull request Nov 21, 2024

How do i connect omniparser to a program of mine #79

Open

techsparkling mentioned this pull request Dec 5, 2024

Omniparser crashes after processing 7 images. #99

Open

Merge branch 'master' into feat/deploy

f2f8ba8

abrichr mentioned this pull request Jan 20, 2025

Feat/deploy deps OpenAdaptAI/OmniParser#4

Open

n-sviridenko mentioned this pull request Jan 21, 2025

Improving performance #118

Closed

Merge branch 'master' into feat/deploy

1cf8c6f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Dockerfile and client.py; deploy to EC2 on AWS via Github Actions#52

Add Dockerfile and client.py; deploy to EC2 on AWS via Github Actions#52
abrichr wants to merge 39 commits intomicrosoft:masterfrom
OpenAdaptAI:feat/deploy

abrichr commented Oct 28, 2024 •

edited

Loading

Uh oh!

abrichr commented Oct 29, 2024

Uh oh!

yadong-lu commented Nov 1, 2024 •

edited

Loading

Uh oh!

abrichr commented Nov 1, 2024 •

edited

Loading

Uh oh!

n-sviridenko commented Jan 21, 2025

Uh oh!

abrichr commented Feb 18, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

abrichr commented Oct 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

🚀 Docker Quick Start

Usage Instructions

Example output

Uh oh!

abrichr commented Oct 29, 2024

Uh oh!

yadong-lu commented Nov 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

abrichr commented Nov 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

n-sviridenko commented Jan 21, 2025

Uh oh!

abrichr commented Feb 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

abrichr commented Oct 28, 2024 •

edited

Loading

yadong-lu commented Nov 1, 2024 •

edited

Loading

abrichr commented Nov 1, 2024 •

edited

Loading

abrichr commented Feb 18, 2025 •

edited

Loading