>

Small LLMs: Unleashing the Power of Conversational AI for Small Devices

Data
|
BuildAI

This presentation will discuss an effective data distillation approach for finetuning small language models for conversational tasks.

Distilling conversational skills into Small Language Models (SLMs) with approximately 1 billion parameters presents significant challenges. Firstly, SLMs have limited capacity in their model parameters to learn extensive knowledge compared to larger models. Secondly, high-quality conversational datasets are often scarce, small, and domain-specific. Addressing these challenges, we introduce a novel data distillation framework named CoDi (short for Conversational Distillation, pronounced “Cody”). In this talk, I will present our CoDy framework to synthesize large-scale, assistant-style datasets in a steerable and diverse manner. I will then discuss comparison of SLMs trained with CoDi-synthesized data to models trained on human-annotated data for conversational grounded reasoning task.

Event Details

Time & Duration:

1:30
20 min

In-Person Location:

Boole Room

Watch online:

Featured Speaker

AI Research Scientist

Kanika is a seasoned professional with a decade of practical experience in the field of AI. She focus on research, development, and implementation of AI models to address complex issues, such as content safety, user behavior modeling, and recommendation systems. As a Research Scientist at Meta Reality Labs, she is pioneering conversational AI projects for […]

Other Speakers Sessions