CAPO: Cost-Aware Prompt Optimization \w Tom Zehle

LLMs have shown to be capable of solving a wide range of tasks, yet their performance is surprisingly sensitive to prompt formulation. Manually tuning prompts can be time-consuming, and we lack clear guidelines on what works well. This makes automated prompt optimization an important tool for improving performance in a systematic and efficient way. In this talk, we introduce CAPO (Cost-Aware Prompt Optimization), a method that finds prompts achieving high performance while keeping costs low. CAPO has been shown to outperform existing prompt optimization techniques across various benchmarks, achieving substantial improvements such as a 25%p accuracy boost on GSM8K using a Llama-3.3-70B. To make these advances accessible to the broader community, we have developed Promptolution, an open-source framework for implementing and experimenting with prompt optimization approaches.
More info: https://munich-nlp.com/events/zehle-capo-cost-aware-prompt-optimization/