icon Takin-VC: Zero-shot Voice Conversion via Jointly Hybrid Content and Memory-Augmented Context-Aware Timbre Modeling

Abstract: Expressive zero-shot voice conversion (VC) is a critical and challenging task that aims to transform the source timbre into an arbitrary unseen speaker while preserving the original content and expressive qualities. Despite recent progress in zero-shot VC, there remains considerable potential for improvements in speaker similarity and speech naturalness. Moreover, existing zero-shot VC systems struggle to fully reproduce paralinguistic information in highly expressive speech, such as breathing, crying, and emotional nuances, limiting their practical applicability. To address these issues, we propose Takin-VC, a novel expressive zero-shot VC framework via adaptive hybrid content encoding and memory-augmented context-aware timbre modeling. Specifically, we introduce an innovative hybrid content encoder that incorporates an adaptive fusion module, capable of effectively integrating quantized features of the pre-trained WavLM and HybridFormer in an implicit manner, so as to extract precise linguistic features while enriching paralinguistic elements. For timbre modeling, we propose advanced memory-augmented and context-aware modules to generate high-quality target timbre features and fused representations that seamlessly align source content with target timbre. To enhance real-time performance, we advocate a conditional flow matching model to reconstruct the Mel-spectrogram of the source speech. Experimental results show that our Takin-VC consistently surpasses state-of-the-art VC systems, achieving notable improvements in terms of speech naturalness, speech expressiveness, and speaker similarity, while offering enhanced inference speed.

Overview of Takin VC

An overall summary of Takin VC model.

Compare with baselines

Source Prompt DiffVC NS2VC VALLE-VC SEFVC Takin-VC










































Paralinguistic Zero-shot Voice Conversion

Source Prompt PPG Generated PPG+WavLM Generated

Laughing




Inhaling Sound




Hush Sound & Cry



Samples of IEMOCAP Dataset
Source Prompt Paralinguistic Tag PPG+WavLM Generated


The sound of clearing the throat.



The sound of clearing the throat.



Hum Sound.



Tail Drag Sound.

Zero-Shot Voice Conversion Across Genders and Languages

Type Source Prompt Generated
[CN] Male-to-Male


[CN] Male-to-Female


[CN] Female-to-Female


[CN] Female-to-Male


[EN] Male-to-Male


[EN] Female-to-Female