DevMan

An example of a commit diff in a merge request
The corresponding code review for the merge request
Example of some pipeline jobs running

Inspiration

There has always been a surplus of pull requests with a shortage of reliable and efficient reviewers. We want to change that.

What it does

Our tool goes through all open pull requests in a repository and provides expert level AI generated tips and comments to gain insight into better coding practices.

How we built it

In Python, we used a locally run llama.cpp instance and interfaced with the GitLab API to go through all new pull requests to provide feedback.

Challenges we ran into

We ran into difficulties prompt engineering to output relevant and consistent feedback but with some experimentation/tinkering, we were able to make a reliable system. We were also able to make use of configuring the AI's grammar to fine tune the format of responses to be how we want.

Accomplishments that we're proud of

Making a full fledged code reviewer that responds relevantly. The relevancy was the hard part, because a lot of times there isn't enough context in the commits, but we did a pretty good job.

What we learned

Specific fine tuning of prompts has much more of an effect on the quality of response than one might think.

What's next for DevMan

We plan to add more tools for repositories, especially those making use of locally sourced AI in which the prompt and grammar can be engineered to be specific for the task at hand. We also want to finish implementing our dynamic review environments feature, where a live instance of the website is deployed when a pull request is made.