2020年4月12日 (日) 12:14的最后版本

Introduction

Speech signals involve complex factors, each contributing in an unknown and secrete way. Recent developed deep learning methods have built up some interesting tools for discovering these latent factors. These tools include various unsupervised models such as VAE, GAN, supervised learning methods such as multi-task learning, knowledge distillation, etc. These tools allow us to decipher secretes of speech signal, based on big data, rather than hypothesis.

These will lead to an unprecedented breakthrough in speech information processing. Some of the signals for this breakthrough includes:

• In speaker recognition, speaker factors can be learned within a very small speech segment.
• In speech synthesis, speaking styles can be learned as latent variables and discovered in an unsupervised way, and speaker factors can be used to change the speaker trait.
• In speech recognition, learning multiple tasks in a collaborative way has shown to be successful.

In previous studies (Phase 1), we have found that using cascade learning, speech signals can be factorized into content, speaker and emotion at the frame level. In this Phase 2, we will try to answer the following questions:

• Can we factorize speech signals in an unsupervised way?
• How supervised and unsupervised factorizations are integrated?
• How to deal with language discrepancy in factorization?
• How to discover optimal factorization architectures?

People

Dong Wang, Yunqi Cai, Haoran Sun, Zhiyuan Tang, Lantian Li

Research direction

Basic research

• Collaborative learning with AutoML
• VAE/dVAE factorization
• Supervised VAE for factorization
• ASR + TTS cycle training

Applied reseach

• Pretraining for ASR, SID, EMD (BERT in speech)
• Low-resource ASR, TTS
• Signal compression, cleaning up, etc.

Related publications

1. Yang Zhang and Lantian Li and Dong Wang, "VAE-based regularization for deep speaker embedding", Interspeech 2019
2. Lantian Li, Yixiang Chen, Ying Shi, Zhiyuan Tang, and Dong Wang, “Deep speaker feature learning for text-independent speaker verification,”, Interspeech 2017.
3. Lantian Li, Dong Wang, Yixiang Chen, Ying Shing, Zhiyuan Tang, http://wangd.cslt.org/public/pdf/spkfact.pdf
4. Lantian Li, Zhiyuan Tang, Dong Wang, FULL-INFO TRAINING FOR DEEP SPEAKER FEATURE LEARNING, http://wangd.cslt.org/public/pdf/mlspk.pdf
5. Zhiyuan Thang, Lantian Li, Dong Wang, Ravi Vipperla "Collaborative Joint Training with Multi-task Recurrent Model for Speech and Speaker Recognition", IEEE Trans. on Audio, Speech and Language Processing, vol. 25, no.3, March 2017.
6. Dong Wang,Lantian Li,Ying Shi,Yixiang Chen,Zhiyuan Tang., "Deep Factorization for Speech Signal", https://arxiv.org/abs/1706.01777

Talks

• Back to Matrix (Dong Wang, 2019/11/02)
• Nolinear_Flow (Yunqi Cai, 2019/11/02)
• Flow_introduction (Yunqi Cai, 2019/11/04)
• Previous_experiments (Haoran Sun, 2019/11/02)

Further reading

Old

1. Barlow, Unsupervised learning, Neural computation, 1989. [1]

1. Geoffrey E. Hinton， Terrence Joseph Sejnowski, "Unsupervised Learning: Foundations of Neural Computation", MIT Press, 1999[2]Associated review from DL Wang [3]

1. Anthony J. Bell and Terrence I. Sejnowski, "An Information-Maximization Approach to Blind Separation and Blind Deconvolution", Neural computation 1995.[4]

1. Gustavo Deco and Wilfried Brauer , "Higher Order Statistical Decorrelation without Information Loss", NIPS 1994. [5]

1. Aapo Hyvarinen and Petteri Pajunen, "Nonlinear independent component analysis: Existence and uniqueness results", Neural networks, 1999. [6]

Linguistics

1. Hiroya Fujisaki, “Prosody, models, and spontaneous speech,” in Computing prosody, pp. 27–42. Springer,1997.

ML

1. Laurent Dinh David Krueger Yoshua Bengio,"NICE: NON-LINEAR INDEPENDENT COMPONENTS ESTIMATION", ICLR 2015 [7]
2. Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio, "DENSITY ESTIMATION USING REAL NVP", ICLR 2017 [8]

1. Kingma, et al., "Auto-encoding variational Bayes". 2014.
2. Danilo Jimenez Rezende et al., "Variational Inference with Normalizing Flows", 2016.
3. Kingma et al., "Improving Variational Inference with Inverse Autoregressive Flow", 2016.
4. Oord, "Neural Discrete Representation Learning", 2017.
5. Kingma, "Glow: Generative Flow with Invertible 1×1 Convolutions", 2018.
6. Goodfellow et al., "Generative adversarial nets", 2014.
7. Zhu et al., "Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks", 2017.
8. Chen et al., "Infogan: Interpretable representation learning by information maximizing generative adversarial nets", 2016.
9. Hu et al., "On unifying deep generative models", 2017.
10. Makhzani, "Adversarial Autoencoders", 2015.
11. Locatello et al., "Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations", 2019.

ASR

1. Chan et al., "Listen, attend and spell: A neural network for large vocabulary conversational speech recognition", 2016.
2. Prabhavalkar et al., "A Comparison of Sequence-to-Sequence Models for Speech Recognition", 2017.
3. Chiu et al., "State-of-the-art Speech Recognition With Sequence-to-Sequence Models", 2018.
4. Pratap, "wav2letter++: The Fastest Open-source Speech Recognition System", 2018
5. Ren et al., "Almost Unsupervised Text to Speech and Automatic Speech Recognition", 2019
6. Tsai et al., "Learning Factorized Multimodal Representations", 2019

SID

1. E. Variani, X. Lei, E. McDermott, I. Lopez Moreno, and J. Gonzalez-Dominguez, “Deep neural networks for small footprint text-dependent speaker verification,”2014.
2. G. Heigold, I. Moreno, S. Bengio, and N. Shazeer, “End-to-end textdependent speaker verification,” 2016.

TTS

1. Wang, et al., "Tacotron: A fully end-to-end text-to-speech synthesis model." CoRR, abs/1703.10135, 2017.
2. van den Oord, et al., "Parallel WaveNet: Fast high-fidelity speech synthesis.", CoRR, abs/1711.10433, 2017.
3. van den Oord, et al., "WaveNet: A generative model for raw audio". CoRR, abs/1609.03499, 2016a
4. Nal Kalchbrenner et al., "Efficient Neural Audio Synthesis", 2018 (WaveRNN)
5. Hsu et al., "Disentangling Correlated Speaker and Noise for Speech Synthesis via Data Augmentation and Adversarial Factorization", NIPS 2018.
6. Wei-Ning Hsu, DISENTANGLING CORRELATED SPEAKER AND NOISE FOR SPEECH SYNTHESIS VIA DATA AUGMENTATION AND ADVERSARIAL FACTORIZATION, ICASSP 2019.

Flow Model

• Overview

• Papamakarios, "Normalizing Flows for Probabilistic Modeling and Inference" [9]

• History

• Whitening:

• Richard M. Johnson. The minimal transformation to orthonormality. Psychometrika, 31: 61–66, 03 1966. [10]
• Friedman, Exploratory projection pursuit, 1987, [11]

• Gaussianization

• Chen et al., Gaussianization, NIPS 2001 [12]
• Tabak et al., Density estimation by dual ascent of the log-likelihood [13]

• Normalization flow

• Tabak et al., A family of non-parametric density estimation, [14] (First composition)
• Rippel et al., High-dimensional probability estimation with deep density model, [15] (first DNN parameterization)
• Laurent Dinh David Krueger Yoshua Bengio,"NICE: NON-LINEAR INDEPENDENT COMPONENTS ESTIMATION", ICLR 2015 [16] (First VP)
• Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio, "DENSITY ESTIMATION USING REAL NVP", ICLR 2017 [17] (First NVP)
• Danilo Jimenez Rezende et al., Variational Inference with Normalizing Flows, [18] (First apply to variational inference)

• Research

2020

1. Structured Output Learning with Conditional Generative Flows, [19]

VAE Model

2014-2015

1. 2014 Auto-Encoding Variational Bayes, [20] (First composition)
2. 2015 The Variational Fair Autoencoder, [21]
3. 2015 Learning Structured Output Representation using Deep Conditional Generative Models, 2.204795282.388480789.1585412171-912935264.1544959736

2017

1. 2017 Variational Lossy Autoencoder, [22]
2. 2017 Grammar Variational Autoencoder, [23]
3. 2017 beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework, [24]
4. 2017 Deep Variational Information Bottleneck, [25]
5. 2017 Understanding disentangling in β-VAE, [26]
6. 2017 Deepcoder: Semi-parametric variational autoencoders for automatic facial action coding, [27]
7. 2017 Hierarchical Variational Autoencoders for Music, [28]
8. 2017 A Classifying Variational Autoencoder with Application to Polyphonic Music Generation, [29]

2018

1. 2018 VAE with a VampPrior, [30]
2. 2018 Hyperspherical Variational Auto-Encoders, [31]
3. 2018 Isolating Sources of Disentanglement in Variational Autoencoders, [32]
4. 2018 Variational Autoencoders Pursue PCA Directions, [33]
5. 2018 Variational Autoencoders for Collaborative Filtering, [34]
6. 2018 Learning Latent Subspaces in Variational Autoencoders, [35]

2019

1. 2019 Structured Disentangled Representations, [36]
2. 2019 Variational Autoencoders with Jointly Optimized Latent Dependency Structure, [37]
3. 2019 Correlated Variational Auto-Encoders, [38]
4. 2019 From Variational to Deterministic Autoencoders, [39]
5. 2019 Variational Laplace Autoencoders, [40]
6. 2019 Disentangling Disentanglement in Variational Autoencoders, [41]
7. 2019 Modeling assumptions and evaluation schemes: On the assessment of deep latent variable models, [42]
8. 2019 Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders, [43]
9. 2019 A Joint Generative Model for Zero-Shot Learning, [44]
10. 2019 A Contrastive Divergence for Combining Variational Inference and MCMC, [45]
11. 2019 Resisting Adversarial Attacks Using Gaussian Mixture Variational Autoencoders, [46]
12. 2019 Hierarchical Decompositional Mixtures of Variational Autoencoders, [47]
13. 2019 Entropic GANs meet VAEs: A Statistical Approach to Compute Sample Likelihoods in GANs, [48]
14. 2019 InfoVAE: Balancing Learning and Inference in Variational Autoencoders, [49]
15. 2019 Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing, [50]
16. 2019 Sparse Multi-Channel Variational Autoencoder for the Joint Analysis of Heterogeneous Data, [51]
17. 2019 Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse, [52]
18. 2019 Learning Hierarchical Priors in VAEs, [53]
19. 2019 CONDITIONALFLOWVARIATIONALAUTOENCODERSFORSTRUCTUREDSEQUENCEPREDICTION [54]

2020

1. 2020 Mixed-curvature Variational Autoencoders, [55]
2. 2020 Variational Autoencoders for Highly Multivariate Spatial Point Processes Intensities, [56]
3. 2020 Mitigating Posterior Collapse in Strongly Conditioned Variational Autoencoders, [57]
4. 2020 When Do Variational Autoencoders Know What They Don't Know?, Skg7VAEKDS

Tools

• VAE: https://jmetzen.github.io/2015-11-27/vae.html
• WaveRNN + VQVAE: https://github.com/mkotha/WaveRNN