• OpenAI API key
• Ruby 3+ or Docker

• Write a prompt using a language flag (e.g. Write a ruby game of tic-tac-toe, Write a bash script to print the current date & hostname, Write a swift program that asks for 2 numbers and returns gcd)

• Only code responses are displayed by default. --verbose flag displays entire api response

• Code responses will trigger a prompt to type a language to evaluate with (or type no to skip & return to prompt)

• If your responses are being cut off, you can increase the max_tokens in the .env file (see model token limits for more info)

• Languages in the table have varying support for writing & running code generated from prompts.
• Note that the language support depends on the language being installed locally and in the users $PATH
• Edit Dockerfile see also: evalgpt.rb

mkdir output

cp examples/.env-example .env # Add your openai api key to the .env file

ruby evalgpt.rb --in path/to/file/with/prompt --out folder/to/write/output/to

See comments in Dockerfile for info on configuring which languages are installed

git clone [email protected]:philipbroadway/evalgpt.git && cd evalgpt
docker build -t evalgpt . && docker run -it evalgpt

• Ruby 3+ installed locally in $PATH
• To generate and eval code in other languages locally the language must be installed locally in $PATH

bundle install

source .env
ruby evalgpt.rb

• Some language interpreters behave differently launching code and running code output may be duplicated or missing. If you eval code and expect a prompt to be shown assume its being shown and enter a value. Investigating way to consistently handle tty io.

** Prompts can take a long time to finish - be patient or use ctrl+c to exit & return to prompt

This is a weekend project for me & contributions are welcome!

Please see contributing guide for more details.