PRR Project

Summary

Improving multimedia information systems that close the gap between human information-needs and the information contained in large repositories of multimodal data, while leveraging well-grounded machine learning methods, is a key research challenge nowadays and in the years to come.The prospective Professor will develop research work in Multimedia Information Systems. In the short-term, it will concentrate on (i) multimodal conversational systems, (ii) planning in large vision and language models, and in the long-term it will address (iii) generative AI methods for multimedia content. A brief description of the expected research activities is given next:Multimodal Conversational Systems – Conversational systems are in its infancy with major breakthroughs that are not yet fully understood. The field is moving quickly, and it will be the root of great changes in the next decade. This natural way of accessing information will turn Information Systems into a completely pervasive commodity. Some of the most urgent research problems that need to be addressed in this area include its visual and linguistic reasoning capabilities, its lack of a real-world model, and its limited capability to keep track of state and sequence. All these capabilities are a corner stone for conversational multimodal systems that aim to guide humans in the physical world.Planning in large vision and language models – Despite the groundbreaking success of large vision and language models, their lack of control, noticeable in its hallucinations, has been a common critique of current LLMs. Hence, being able to control and audit LLMs is critical to many application domains where the safety of people is at risk. For these reasons, one would expect to see LLMs that can be instructed to follow a plan and guide a user through a sequence of actions.Generative AI for Multimedia Content – Automatically generating an image or a video from a text is currently an emerging area that has the potential to disrupt many areas such as video summarization, story illustration, multimodal dialog, and also as a data augmentation tool for other methods. Similarly to LLMs, empirical measures of consistency are still far from being sufficiently reliable with many hallucinations.The prospective Professor will enlarge the educational offer of DEEC/IST in this area. Two possible courses strongly related to the scientific activity to be developed by the prospective Professor are (i) Vision and Natural Language Systems (MSc level) and (ii) Deep Multimodal Information Processing (PhD level).The goal of the first course is to cover the first notions of Vision and Language information processing, addressing (i) learning algorithms, (ii) natural language, and (iii) visual analysis of information. The second course covers (i) multimodal information embeddings, (ii) storytelling and summarization of information streams, and (iii) quality of experience in visual information.This work is strongly related to the activities of DEEC/IST and the Human Language Technology group from INESC-ID. The Human Language Technology group (HLT) includes researchers/faculty from various universities, as well as graduate and post-graduate researchers, totaling around 30 researchers. Their background ranges from Electrical Engineering to Computer Science, and Linguistics. This strongly interdisciplinary group is actively involved in many areas of language research and development, both spoken and written, including, without limitation, speech recognition, speech synthesis, speech coding, speech understanding, audio indexing, multimodal dialogue systems, language and dialect identification, speech-to-speech machine translation, automatic summarization, natural language database interfaces, natural language generation, text simplification, named entity recognition, question answering, among others and in no particular order. The group has a very close cooperation with the spin-off company Voice Interaction, and with Unbabel, launched by a former PhD student.