Prompt robustness in code generation : from surface perturbations to semantic omission and light agent-based repair

Abstract

Context: Large Language Models (LLMs) have become increasingly capable in code generation, yet their performance remains highly sensitive to variations in user prompts. In practical use, prompts often contain imperfections such as typographical errors, alternative wordings, or missing pieces of essential information. These variations influence the model’s reasoning in different ways. Understanding their impact and finding ways to reduce this sensitivity is essential for improving the reliability of AI-assisted coding systems.

Objective: This thesis examines how different types of prompt perturbations affect the robustness of LLMs in code generation tasks, and whether a lightweight dependency-aware agent can mitigate these effects. Method: Using the CodeT5+ model and sanitized MBPP benchmark (200 tasks), this study conducts three types of perturbations: typo, synonym, and semantic omissions, at controlled strengths (0.10,0.25,0.40). A deterministic executionbased evaluation (Pass@1, Pass@5, Pass@10) is applied under identical inference settings. A lightweight dependency-aware agent then restores omitted context and injects targeted library hints before re-evaluation. Results: Typographical errors cause the largest numerical performance drop. Synonym substitutions have minimal impact on the performance. Semantic omission attacks lead to a steady decline. When a lightweight dependency agent is introduced to restore missing semantic cues, Pass@1 improves from approximately 25.5- 26.0 percent to 27.0 percent, while Pass@5 and Pass@10 slightly exceed the clean baseline performance. Conclusion: The effective of the lightweight dependency-aware agent in mitigating omission-related failures suggests a scalable way to bridges the gap between internal self-corrections and costly full agent loops. This study contributes to the development of more robust code generation pipelines build on LLMs by introducing a hierarchical framework for analysing prompt perturbations and by providing a repair module that enables transparent and controllable intervention during inference.