AAAI Tutorial

Thirty-Third AAAI Conference on Artificial Intelligence (AAAI 2019) will take place on January 27-February 1, 2019, Honolulu, Hawaii, USA.

TitleDeep Bayesian and Sequential Learning

Time: 8:30 AM -12:30 PM on January 28

Place: Hilton Hawaiian Village

Description: This tutorial addresses the advances in deep Bayesian and sequential learning for natural language with ubiquitous applications ranging from speech recognition to document summarization, text classification, text segmentation, information extraction, image caption generation, sentence generation, dialogue control, sentiment classification, recommendation system, question answering and machine translation, to name a few. Traditionally, "deep learning" is taken to be a learning process where the inference or optimization is based on the real-valued deterministic model. The "semantic structure" in words, sentences, entities, actions and documents drawn from a large vocabulary may not be well expressed or correctly optimized in mathematical logic or computer programs. The "distribution function" in discrete or continuous latent variable model for natural language may not be properly decomposed or estimated in model inference. This tutorial addresses the fundamentals of statistical models and neural networks, and focus on a series of advanced Bayesian models and deep models including hierarchical Dirichlet process, Chinese restaurant process, hierarchical Pitman-Yor process, Indian buffet process, recurrent neural network, long short-term memory, sequence-to-sequence model, variational auto-encoder, generative adversarial network, attention mechanism, memory-augmented neural network, stochastic neural network, predictive state neural network, policy gradient and reinforcement learning. We present how these models are connected and why they work for a variety of applications on symbolic and complex patterns in natural language. The variational inference and sampling method are formulated to tackle the optimization for complicated models. The word and sentence embeddings, clustering and co-clustering are merged with linguistic and semantic constraints. A series of case studies are presented to tackle different issues in deep Bayesian learning and understanding. At last, we will point out a number of directions and outlooks for future studies.

Organization: The presentation of this tutorial is arranged into five parts. First, we share the current researches on natural language processing, statistical modeling and deep neural network and explain the key issues in deep Bayesian learning for discrete-valued observations and latent semantics. Modern natural language models are introduced to address how data analysis is performed from language processing to semantic learning and memory networking. Secondly, we address a number of Bayesian models to infer hierarchical, thematic and sparse topics from natural language. In the third part, a series of deep models including deep unfolding, Bayesian recurrent neural network (RNN), sequence-to-sequence learning, convolutional neural network, generative adversarial network and variational auto-encoder are introduced. The fourth part illustrates how deep Bayesian learning is developed to infer the recurrent and dilated neural networks for natural language understanding. In particular, the memory network, neural variational learning and Markov RNN are introduced for practical tasks, e.g. speech recognition, reading comprehension, sentence generation, dialogue system, question answering and machine translation. In the final part, we spotlight on some future directions and challenges with big data, heterogeneous condition and dynamic system. In particular, deep learning, structural learning, temporal and spatial modeling, long history representation and stochastic learning are emphasized. (Slides)