From Unimodal to Multimodal Machine Learning

With the increasing amount of various data types, machine learning methods capable of leveraging diverse sources of information have become highly relevant. Deep learning-based approaches have made significant progress in learning from texts and images in recent years. These methods enable simultaneous learning from different types of representations (embeddings). Substantial advancements have also been made in joint learning from different types of spaces. Additionally, other modalities such as sound, physical signals from the environment, and time series-based data have been recently explored. Multimodal machine learning, which involves processing and learning from data across multiple modalities, has opened up new possibilities in a wide range of applications, including speech recognition, natural language processing, and image recognition.

From Unimodal to Multimodal Machine Learning: An Overview gradually introduces the concept of multimodal machine learning, providing readers with the necessary background to understand this type of learning and its implications. Key methods representative of different modalities are described in more detail, aiming to offer an understanding of the peculiarities of various types of data and how multimodal approaches tend to address them (although not yet in some cases). The book examines the implications of multimodal learning in other domains and presents alternative approaches that offer computationally simpler yet still applicable solutions. The final part of the book focuses on intriguing open research problems, making it useful for practitioners who wish to better understand the limitations of existing methods and explore potential research avenues to overcome them

Focuses on combining internal representations in multimodal machine learning Explores different approaches to solving the challenge of combining information Includes an overview of the trends in the field, such as multimodal language models

Autorentext

Blaz Skrlj is a postdoctoral researcher and a research assistant at Jozef Stefan Institute, where he investigates the domain of efficient multimodal machine learning and low-resource machine learning. Blaz completed his PhD in Information and Communication Technologies at the Jozef Stean International Postgraduate School. His work focused on neuro-symbolic machine learning, automated machine learning (AutoML) and representation learning. He authored and co-authored more than fifty research publications, mainly on machine learning and its applications in biomedicine and bioinformatics.

Inhalt

Part .I. Introduction.- Chapter.1.A brief overview of machine learning .- Chapter.2.Data modalities and representation learning.- Part II Unimodal machine learning.- Chapter.3.Learning from text.- Chapter.4.Graph-based methods.- Chapter.5 Computer vision.- Part. III. Multimodal machine learning.- Chapter.6.Multimodal learning.- Part. IV.A look forward.- Chapter.7.Future prospects.