Orca, a Microsoft research project, focuses on developing and refining small language models (LMs) with around 10 billion parameters or fewer. The project’s core objective is to enhance these LMs by imbuing them with advanced reasoning capabilities typically associated with larger models.
Central to Orca’s approach is the concept of imitating the reasoning processes of larger LMs. Through “explanation tuning,” Orca models achieve significant performance gains in challenging zero-shot reasoning benchmarks. For example, Orca surpasses Vicuna-13B by more than 100% in tasks like Big-Bench Hard (BBH) and demonstrates a 42% improvement on AGIEval.
Orca also focuses on optimizing smaller LMs to perform competitively in various tasks, often achieving results comparable to or even better than models 5-10 times their size. These capabilities are rigorously tested in zero-shot scenarios, showcasing the potential of smaller models in practical applications.
