Today, the market is filled with LLM-powered tools, with new and more advanced options emerging almost daily. In this article, we focus on one use case: integrating AI into the software development process to simplify onboarding for new developers.
What is involved in the Onboarding process?
Onboarding a new developer involves a mix of tasks for both the newcomer and the existing team, often making the process hard to standardise or automate. Early days typically include reading documentation, meeting the team, attending sessions, and waiting for access to systems and tools.
The complexity of onboarding can vary by organisation. In large enterprises, developers face vast documentation, sprawling codebases, and rigid processes. In smaller teams, the challenge may be a lack of documentation altogether.
Common Pain Points:
Reading extensive documentation to understand the company, team, and project
Navigating poorly documented or undocumented codebases
Working with unfamiliar languages, tools or architectures
Delays in accessing necessary tools and services
How can AI be used to help with these problems?
Large Language Models (LLMs) have quickly become some of the most widely adopted AI tools in software development. Trained on vast datasets — including code — they are now applied across multiple stages of the development lifecycle.
For developer onboarding, LLMs offer several key advantages:
Simplified access to documentation through summarisation and topic-based search
Interactive learning instead of passive reading
Automated access provisioning by integrating with tools like GitHub and cloud platforms
Real-time code understanding, including syntax explanations, interpreting logic, and answering questions
Popular tools like GitHub Copilot, Claude, Phind, Cody, and Codeium act as AI pair programmers, integrating with IDEs to support developers with suggestions, code insights, and search.
⚠️ Note: The AI tool landscape is evolving rapidly. While this experiment used tools relevant at the time, the core principles and patterns discussed here remain valid.
The Experiment
To evaluate how AI tools can support developer productivity, we focused on a common onboarding challenge: understanding an unfamiliar codebase. This is particularly difficult for:
Developers transitioning between teams or organisations
Freshers with limited exposure to real-world tools and technologies
We selected a moderately sized web application built with Vue.js on the frontend, running on a Node.js backend, hosted on Google Cloud Platform (GCP), and using Firebase for data storage.
The project is fully functional, actively maintained, and follows a standard project structure.
We approached it as a new developer with no prior knowledge of the codebase or tech stack. To assist with onboarding, we used Cody — a free, LLM-powered AI assistant designed to help developers understand and navigate code.
🔧 We used the Cody plugin in VS Code, focusing specifically on its chat feature, which allows users to ask natural language questions about the codebase.
💬 We tested Cody with questions of varying complexity and context to see how well it could support a developer during onboarding.
Note: While Cody includes other development features, this experiment focused solely on its code comprehension capabilities.
🧭 General Repository-Level Questions
We began with high-level questions such as:
What is this project about?
How is the code structured?
Cody successfully interpreted the project context by scanning the available documentation. It provided a clear project summary and outlined the code structure. When asked follow-up questions, it offered deeper insights.
For a developer unfamiliar with Vue.js, this initial support was incredibly useful in understanding the code flow and infrastructure.
➡️ View full set of prompts and responses
📁 File Exploration
As part of our evaluation, we focused on how Cody handles file-level understanding. This included exploring individual files, asking about syntax, structure, imports, and frameworks like Vue.js.
🔍 General File-Level Understanding with “Explain Code”
We selected a few files and asked Cody to explain:
The overall purpose of the file
Variable usage and key data structures
Component layout and how elements are organised
Cody’s “Explain Code” feature was particularly useful — it autogenerates prompts that act as a guide for asking questions. Initially, it offers a high-level overview, and with further prompting, dives deeper into technical detail, explaining syntax and architecture effectively.
⚠️ Limitations in Summarisation
While the explanations were helpful, Cody showed some limitations when summarising entire files. For instance, in one component, it only mentioned the created lifecycle hook—ignoring three others present in the same file.
✅ However, when asked more specific follow-up questions, Cody was able to correct and expand on its response.
➡️ See incomplete summary and follow-up correction
📚 Understanding Imports and Vue.js Concepts
We also tested Cody’s ability to explain:
The purpose of imported libraries
The roles of methods and components used
Basic Vue.js concepts
For someone new to Vue.js, these responses were extremely helpful. Cody offered clear explanations of common imports and their context-specific usage. When prompted, it also suggested external resources for deeper learning.
➡️ View example prompts and responses
🔄 Explaining Code Flow
We then asked Cody to explain the overall flow of the repository. The responses were built on a basic structure of Vue.js combined with how the files are structured in this project. When provided a specific file for context, it was able to identify a few routes in the project.
➡️ View full conversation here
📌 Importance of Context during prompt
When selecting a code snippet and asking where the data was loaded from:
With only repo-level context, Cody returned generic, assumption-based answers
When given the specific file, it correctly traced the data-fetching function
➡️ Example showing difference in context
🎨 Tracing CSS Classes
Cody struggled to trace CSS class definitions:
It succeeded at times, if the style file was open in the editor
Otherwise, it provided generic explanations based on typical usage
🧠 Method Tracing in Vue Files
We tested Cody’s ability to trace method usage:
In some cases, it returned initially incorrect answers
But it often corrected itself upon rephrasing or providing more context
When asked to find references to a known method, it successfully identified some instances
Notably, Cody didn’t hallucinate; it stuck to what’s present in the project or what could be reasonably inferred.
➡️ View the method tracing example
🗺️ Analysing Routing Configurations
Understanding routing is crucial for new developers. However, this project had both legacy and new routing structures, with duplicate filenames in different paths.
Cody struggled to correctly resolve imports and map components to paths. Despite this, it offered a reasonable fallback explanation based on router files — and notably, did not hallucinate.
➡️ See routing config exploration
🏢 Understanding Business Context from Code
We asked Cody about business-specific logic in the codebase.
Without explicit context, responses were vague and based on web conventions
However, when the business context was provided within the code, Cody was able to recognise and respond appropriately
➡️ Example of business context prompt
How useful is it really…?
As part of this experiment, we focused solely on testing Cody’s chat feature, as our primary goal was to evaluate its effectiveness in supporting developer onboarding.
Since Cody can use the entire repository as context, it proved quite useful for answering high-level questions. Here’s what it did well:
Project summarisation: Provided both general and technical overviews of the project.
Technology insights: Identified the technologies used, along with their versions and configurations.
Project structure: Accurately outlined the structure and infrastructure of the codebase.
In cases where Cody couldn’t locate specific code blocks or references within the repository:
It offered generalised responses based on the tech stack.
Notably, it did not hallucinate or fabricate information, instead defaulting to safe and informed assumptions.
Limitations
Cody — and similar AI tools — also come with certain limitations. Below are some of the key issues we observed during the experiment:
🔹 1. Contextual Understanding
One of the major limitations was handling business context. When we asked questions about specific elements related to the business logic, Cody often failed to provide accurate answers. To overcome this, we had to manually supply context to help it interpret the relevant sections of code.
🔹 2. Difficulty Tracing Method Definitions
Cody struggled to trace methods to their definitions, especially when multiple files were involved. This may be specific to the tech stack we used, but we noticed a general decline in accuracy as the code path became more complex.
For new developers trying to follow the flow of code across files, this could be a limitation. While the tool can often point you in the right direction, it doesn’t always produce precise results.
🔹 3. Inconsistent Support for Certain File Types
Cody had trouble understanding parts of the codebase such as CSS definitions. These areas remained opaque or poorly interpreted, limiting its usefulness.
🔹 4. Inaccurate Route Tracing
Even when route configurations were defined in a single file, Cody often gave inaccurate mappings between pages and their paths. This made it unreliable for understanding the routing setup in detail.
🔹 5. Performance and Stability Issues
On several occasions, Cody crashed or stopped responding mid-answer, interrupting the workflow.
Conclusion
In conclusion, LLM-powered tools like Cody can significantly enhance developer productivity, particularly during the onboarding process. Instead of spending hours sifting through extensive documentation and navigating an unfamiliar codebase, developers can rely on these tools to:
Summarise documentation and code effectively
Explain new concepts and unfamiliar code snippets
Provide quick answers to contextual questions about the project
For new developers, tools like Cody can help them quickly understand the landscape and become productive sooner. Despite some limitations, they remain a valuable aid in streamlining the onboarding experience.
