Positive Highlights
Below, we review and largely reinforce the positive highlights which drove our initial thesis to invest in CodiumAI:
- Software of questionable quality costs companies trillions of dollars. Shifting testing left can significantly alleviate this burden: The cost of fixing software errors compounds as errors move through the SDLC. Preventing software errors is better than fixing them in production, and the best way to prevent errors is to test code often and thoroughly. Reports show that over 80% of software issues lie within the SDLC, with errors in production costing 30x more to fix than in development. We expect developers to seek innovative code testing and integrity tools and believe CodiumAI is well positioned to be that trusted party.
Commentary: Since this memo’s writing, the rise in popularity of AI assistants (i.e., Copilot, ChatGPT, and CodeGPT) has propelled developers to write more code faster. While assistants have delivered productivity benefits, they have also increased the risk of error-filled code slipping into production. One of the reasons for this is the misalignment between developers’ intent and what the LLM assumes before generating code. The misconception that code completion and generation tools are bug-free gives developers false confidence in the accuracy and reliability of their code, increasing the need for code-level testing. Unlike generalist code generation tools, CodiumAI reviews and understands code structure and desired functionality, considering metadata such as code comments while intelligently interacting with the developer via prompts in the IDE.
- Existing software testing solutions are outdated and ineffective. Code Integrity is a new paradigm that reimagines unit testing, code analysis, and code coverage: While basic testing frameworks exist, developers are responsible for manually implementing their test logic. Our conversations with them and their team leads validate the difficulty of writing effective and accurate unit tests using existing solutions. Moreover, developers spend >20% of their time writing tests, which is time not spent pushing new software. CodiumAI empowers developers to generate meaningful interactive test suites within their IDE by analyzing source code, docstring, and comments, equipping developers with code correctness and quality-checking capabilities at a level previously unavailable in an automated (i.e., generative) way.
Commentary: Today, we define Code Integrity as (a) the code correctness-checking processes accompanied by (b) the metrics to measure the completeness of these code correctness-checking processes. While automated unit tests are a powerful method and wedge into (a), CodiumAI’s approach expands across static/dynamic code analysis, code review automation, and even comparing code behavior to written product specifications – further distinguishing it from existing testing solutions. We have also observed increasing demand from CTOs and team leads for improved code coverage measures across the organization through a centrally governed platform. Since our investment, it has become clear that developers and executives are demanding the tooling, automation, and unification of both (a) and (b). Expanding their enterprise offering, CodiumAI’s platform may include collaboration tools, test data management, CI/CD integration, auto-fixing of bugs, code improvement suggestions, and eventually, the enablement of next-generation, test-driven development.
- Breakthrough generative technology paired with a world-class team: Until recently, LSTMs have been the state-of-the-art model architecture for code generation. The advent of the Transformer architecture unlocks tremendous scale and generalizability of Large Language Models (LLMs), delivering the most powerful code generation yet. Examples of recently trained code generation LLMs include (1) Deepmind’s AlphaCode, (2) OpenAI’s Codex, and (3) Facebook’s InCoder. While this technology will be impactful, leveraging it effectively is challenging and requires a combination of deep academic and commercial experience. Harnessing both of these, CodiumAI’s founding team are serial entrepreneurs who previously founded and exited startups and led product and R&D teams at leading organizations like Alibaba Cloud. The team started CodiumAI to address the chronic pain they dealt with throughout their careers: verifying and validating code.
Commentary: Improvements in the quality and accessibility of LLMs have been remarkable. Model layer providers such as OpenAI, Stability, Cohere, Adept, and AI21 have innovated across text, vision, and audio. While CodiumAI is incorporating existing state-of-the-art LLMs, the team is executing rapidly on training their own LLM and system, TestGPT. TestGPT specializes in code testing (including designated integrations and data digestion capabilities) and is trained on a large, proprietary test-related dataset. Aside from the LLM itself, CodiumAI’s team has built a significant engineering stack and designed a product experience (more below) that significantly supplements their model.
