In software development, code review is an essential step that ensures the quality and readability of the codebase. To improve the code review process, the frontend platform team at Adjust explored the possibility of using ChatGPT as a pull request (PR) review assistant and built a proof of concept.
This article will discuss the idea, exploration, implementation, and challenges of using ChatGPT within our ecosystem.
Image from Unsplash
💡Idea
As a Frontend platform team, constantly improving our internal teams’ development process and experience is one of our goals. At Adjust, for the frontend projects, it is mandatory that at least one developer reviews the code and approves it to merge the PR. However, it can be time-consuming and requires team members to read, think, and evaluate new features. We could use ChatGPT as a PR review assistant for the primary reviewer to simplify and speed up the code review process.
🔭 Exploration
As a first step, we explored all the possible ways to implement this. After some design discussions, we concluded that implementing this system within our CI/CD pipeline would be the best choice, as we want to perform the PR review process on Github right after the developer creates a PR.
Then, we explored if there are already existing solutions on Github marketplace. Many did things like summarizing code changes, commenting on all the changed files, and not providing flexibility to configure the action to our use cases. After some exploration, we found one action closer to what we sought. But the issue with it was it doesn’t work anymore with new API changes of OpenAI ChatGPT, and the project was archived.
🛠️ Implementation
We forked that GitHub action, modified the code to support the new API changes of chatGPT and made it work (Woot, woot! 🎉). We also improved it by making it more flexible by making it accept model and prompt count and updating the prompts.
We used the OpenAI playground to test the different prompts and came up with the best one that suited our needs.
Of course, due to security reasons, we cannot use or test them on our actual GitHub repository under Adjust org. We tested it on sample GitHub public projects, and the results were satisfactory. 🤷♂️
💪 Challenges
We encountered several challenges when we were working on this project. To highlight some of them:
Context Limitation: Open AI models have limitations on the context length of 4,096 tokens. Initially, when we sent the entire PR all at once to chat GPT, it exceeded the limit. We overcame this problem by extracting only the code diff and sending it to Chat GPT for review.
Server Downtime: During our development phase, the Open AI system used to go down regularly, which was a blocker for us sometimes.
Prompt Engineering: Prompt engineering is the crucial aspect of utilizing LLMs effectively and is a powerful tool for customizing the interactions with ChatGPT. Getting the right prompt that can handle all our use cases is tricky. After trial and error, we designed a prompt that works well for all our use cases.
🧐 What next?
The primary purpose is to improve the experience of the code reviews.
We want to start with running a pilot to review the Frontend interview home tasks as an assistant to the primary reviewer. It will help do the home task review quickly and provide the feedback to the candidate early.
Further, we plan to use ChatGPT to help write unit tests for each newly created PR.
Can ChatGPT write code for the tasks that we work on in our day-to-day lives? Well, let’s see. 🤔
🧑🚀 Authors
Meet the mighty little Frontend platform team: