This presentation will discuss an effective data distillation approach for finetuning small language models for conversational tasks.
Distilling conversational skills into Small Language Models (SLMs) with approximately 1 billion parameters presents significant challenges. Firstly, SLMs have limited capacity in their model parameters to learn extensive knowledge compared to larger models. Secondly, high-quality conversational datasets are often scarce, small, and domain-specific. Addressing these challenges, we introduce a novel data distillation framework named CoDi (short for Conversational Distillation, pronounced “Cody”). In this talk, I will present our CoDy framework to synthesize large-scale, assistant-style datasets in a steerable and diverse manner. I will then discuss comparison of SLMs trained with CoDi-synthesized data to models trained on human-annotated data for conversational grounded reasoning task.
