Multimodal research is a scientific approach that combines different methods and types of data to gain broader and deeper insights into complex phenomena. This research approach utilises multiple modalities, which means that different types of data and methods are integrated to illuminate a research topic from different perspectives.
Integration of different data types
Quantitative data: Number-based data that can be statistically analysed.
Qualitative data: Descriptive data that is available in the form of text, images or videos and is analysed descriptively.
Use of different research methods
Experimental methods: Controlled experiments to investigate causal relationships.
Observation methods: Direct observation of behaviour and interactions in natural environments.
Survey methods: Surveys, interviews and questionnaires to gather opinions and experiences.
Integration of different disciplines
Multimodal research is often interdisciplinary and combines findings from different scientific fields such as psychology, sociology, computer science, neuroscience, and others. Typical application areas are:
Human Factors research
Usability and UX research
Gaming research
Market research
Ergonomics
Neuromarketing
and more...
Advantages of multimodal research
Holistic understanding: By combining different data sources and methods, researchers can develop a more comprehensive and nuanced understanding of complex phenomena.
Increased validity: The use of multiple methods and data sources can increase the validity of the research results by combining the strengths of the individual approaches.
Broader perspectives: Multimodal research makes it possible to look at issues from different scientific perspectives and thus gain new insights.
Examples of multimodal research
Educational research: Analysis of student performance by combining standardised test results (quantitative) and interviews with teachers and students (qualitative) to obtain a comprehensive picture of educational quality.
Medical research: Investigation of disease progression by combining genetic data, clinical test results and patient interviews.
Cognitive neuroscience: Research into brain processes using a combination of imaging techniques (e.g. fMRI, EEG), behavioural studies and psychometric tests.
Technology development: Development and improvement of human-machine interactions through the analysis of user data, observation of user interactions and qualitative feedback from users.
Challenges of multimodal research
Complexity: The integration of different methods and data types requires careful planning and coordination.
Data management: Handling large and heterogeneous volumes of data places high demands on data processing and analysis.
Interdisciplinary collaboration: Often requires collaboration between researchers from different disciplines, which can pose organisational and communication challenges.
Overall, multimodal research is a powerful approach for capturing the complexity of the real world and gaining well-founded, comprehensive insights.
Mangold Observation Studio is a groundbreaking new software product for multimodal data collection and analysis.
Record all data synchronously: A core functionality of Mangold Observation Studio is the time-synchronous recording of multimodal data, such as video, physiology, computer screens and more, allowing you to conduct qualitative and quantitative observational studies in a wide variety of applications.
Easy to use: The system is literally designed as a one-push-button solution, yet has a wide range of features and configuration options.
Live viewing: All recorded data can be viewed live by all stakeholders, such as test managers, students, engineers, designers, marketing managers, or other interested parties.
Take advantage of our decades of experience in observational research.
Multimodal Data Acquisition in Human Factors Research
What is Human Factors Research?
Human factors research is concerned with human behaviour and processes, including human capabilities and limitations, in order to develop designs and processes that best meet human needs.
Some of the most pressing questions that researchers in the field of human factors are trying to answer:
How can we design technologies and systems so that they are more user-friendly and intuitive?
How can we improve the safety and efficiency of complex systems such as aviation, healthcare and transport?
How can we design workplaces that promote well-being, productivity and creativity?
How can we consider the impact of new technologies such as artificial intelligence, automation and virtual reality on human performance and behaviour?
How can we understand and account for the impact of age, culture and individual differences on human performance and behaviour in different contexts?
How can we promote effective communication and teamwork in high-stress, high-stakes environments?
How can we design training and education programmes that are effective, efficient and engaging?
How can we evaluate the effectiveness of measures and technologies to improve human performance and behaviour?
...
In short, human factors research is about how people interact with technology, systems and environments.
These issues relate to practical challenges and have an impact on improving human well-being, productivity and safety.
Research in the field of human factors can lead to practical solutions and innovations that benefit individuals, organisations and society as a whole.
Synchronize Multiple Data Modalities for Human Factors Research
Synchronized recording of all this data is the main challenge, but a full-featured Mangold Human Factors Lab makes it possible and manageable.
Another often underestimated but major challenge is to playback and review all the collected data streams in a synchronized manner to find relevant and viable results.
In addition, according to your specific needs, it will include various hardware devices like cameras, bio-physiological sensors, and maybe an eye tracker. These software and hardware packages work together hand in hand to integrate different data modalities.
Integrate Multiple Data Modalities with a Mangold Human Factors Research Lab
It records multiple audio and video sources in sync, accepts markers from different sources, functions as a debriefing station, and can send a signal to third-party recorders that allow us to insert a synchronization entry.
Synchronization Between Human Factors Lab Components
Mangold VideoSyncPro Studio can send a 'recording started' signal to external recorders in several ways: through your network via UDP, by using a USB switch in combination with a dedicated cable to the external recorder, by a keyboard signal for applications running on the same computer as VideoSyncPro Studio, or by a binary LTP port signal with a dedicated cable.
It is also possible to remotely control VideoSyncPro Studio from an external application using UDP signals. This method allows you, for example, to start recording externally if this external application can send custom UDP commands.
Lastly, activating presets to position one or more cameras automatically inserts markers into the recording log file for visual reference and identification of a change in position. External events captured from or generated by external devices or third-party software products can also be automatically inserted as markers or imported afterward.
Add Observations to Measurements for Human Factors Research
Each Mangold Human Factors Lab comes with the Mangold Remote Observer. This add-on allows you to supervise and fully remote control a recording session on your Mangold VideoSyncPro Studio recording station.
Using a handheld device, you can start and stop recordings, log markers during the session, and use presets to control your cameras.
Multiple Remote Observers can connect to the same sessions to log independent observations from another perspective or while each observer focuses on a different participant.
After the session, you can enrich your live observations with further details in Mangold VideoSyncPro Studio using its Marker & Debriefing functions.
Alternatively, you can transfer your live-markers to Mangold INTERACT to add fine-granulated observations with perfect timing and maybe using hierarchical Coding Systems if required. Mangold INTERACT allows you to go into much more detail and enables close examination of all sources simultaneously.
Benefits of Integrating Observations and Sensor-based Measurements
For starters, measured values can be used to automatically generate Events into Mangold INTERACT, based on values in a specific channel.
This functionality allows you to watch exactly those sequences in the video, or examine what happened in the last so many seconds before a specific drop or increase of values occurred, etc.
For example, whenever the participant's heart rate rises above 90 bpm, you want to examine a) if there are any visual signs of this increased heart rate and b) what happened in the last, e.g. 30 seconds before this threshold was breached.
Secondly, you can quickly transfer the observation codes you used for your observations directly into the raw measured data. This function applies all selected observation codes assigned to specific time frames to all values measured during exactly these time frames.
The combination of measured values and observational data on this raw data level allows you to calculate totals, averages, standard deviations, and more over the measured values per observed behavior right away or process the combined values and observations through special statistics applications like R.
Analyze Multidimensional Data from a Human Factors Research Lab
For a Mangold Human Factors Research lab, the software module Mangold INTERACT is the central application for multidimensional data analysis. With its external data visualization module Mangold DataView we create dynamic charts with the captured numeric data running in sync with your videos.
Combining events that were automatically generated based on measured values with manually logged observations and applying the available time-related data mining functions can really help you to get the best out of all data collected.
With Mangold INTERACT, you can, among other things, determine contingencies, find specific co-occurrences, and generate coding segments for the overlapping of all information or run the sequence analysis routine.
Manually adapt any offset with millisecond accuracy for all data streams, including video!
As discussed before, it is crucial to synchronize all your data streams for in-depth analysis. Ideally, this synchronization is covered during recording and the import process.
Still, Mangold INTERACT and its external data visualization module Mangold DataView provide several mechanisms to manually synchronize your observational data, videos, and externally recorded data values. You can adapt the required offset with millisecond accuracy for any data stream, including video.
The "Dirty Dozen" in Human Factors Research
The "Dirty Dozen" is a list developed by the aviation industry that summarises twelve common sources of human error that can lead to safety-related incidents and accidents. It was originally developed by Gordon Dupont as part of a programme run by the Canadian aviation authority Transport Canada and has since established itself in many industries as an important training tool for promoting a culture of safety.
The "Dirty Dozen" of Human Factors was originally developed for aviation, but is also relevant in many other areas.
Here is a brief overview of the "Dirty Dozen":
Lack of communication: Lack of or unclear communication can lead to misunderstandings and critical information being lost, jeopardising safety.
Fatigue: Fatigue reduces concentration and judgement, which increases the likelihood of errors.
Lack of teamwork: Poor teamwork can lead to inefficient work processes and misunderstandings that jeopardise safety.
Distraction: Unforeseen disruptions or interruptions can take focus away from important tasks, which can lead to errors.
Lack of resources: Lack of or insufficient resources, such as tools, time or personnel, can lead to improvised solutions and potentially unsafe working practices.
Pressure: Time or performance pressure can lead to decisions being made hastily, increasing the likelihood of errors.
Complacency: Overconfidence in one's own abilities or in best practice can lead to underestimating risks and ignoring safety precautions.
Lack of reassurance: Failure to verify important information or actions can lead to errors going undetected and escalating.
Poor ergonomics: Inappropriate working conditions or equipment can lead to physical stress, errors and injuries.
Lack of knowledge: Insufficient knowledge or training about procedures, equipment or situations can lead to employees making unsafe decisions.
Lack of attention: When people rely on routine tasks, vigilance can wane, causing signs of problems to be missed.
Standards: Informal pressure to accept ways of working that do not conform to prescribed procedures can lead to unsafe practices and ultimately incidents.
Why are the "Dirty Dozen" important?
The "Dirty Dozen" help to raise awareness of human factors that can affect safety in different working environments. They serve as a checklist and training tool to identify and minimise risks and promote a strong safety culture. By understanding and addressing these twelve hazards, we can prevent accidents and make the working environment safer.
Human Factors Research Lab and Multimodal Data
In order to answer these complex questions, a lot of different data needs to be collected.
Measuring the various signals from the human body in response to the different stimuli from our environment is one part of this data, observing the participants during their daily work is another part.
Sometimes signals from devices we interact with, such as computers, gaze data, simulation stations, etc., are also included in the data pool.
By measuring and integrating these physiological signals in combination with real-life observations, we can better understand conscious and unconscious reactions.
Typical modalities that may be of interest for a human research study are:
Recorded videos
Screen recordings that are recorded synchronously with other cameras
Markers and notes collected during the recording
Behavioural events recorded by a human observer
Physiological signals recorded by third-party data collection systems
Gaze data from eye trackers
Data from other programmes that can provide text files with time information or a fixed sampling rate
Keystrokes, mouse clicks and other events recorded by Mangold LogSquare, for example
Facial expressions, recorded manually or recognised automatically by an AI
The time-synchronized integration of the various collected data is essential for a comprehensive understanding of these processes in their different contexts.
For this reason, a Mangold Human Factors Research Lab usually contains various devices for recording video and audio signals, bio-psychological measurements, EEG data, eye-tracking data and stimulus generators.
Multimodal Data Acquisition in Usability and User Experience Research
Mangold Observation Studio can be used to evaluate usability and user experience in various scenarios.
What is User Experience and Usability?
User experience (UX) and usability are concerned with designing and optimising the interaction between people and digital products in order to ensure a positive, intuitive and efficient user experience.
Usability focuses on functionality and user-friendliness - in other words, how easy and efficient a product is to use.
User experience (UX), on the other hand, looks at the entire user experience, including usability, but also other factors such as emotions, expectations and overall satisfaction.
While usability is a part of user experience, user experience is the broader term that includes functionality as well as the emotional and aesthetic aspects of user interaction with a product.
Some of the key questions that researchers in the field of UX and usability are trying to answer:
How can we design digital products and systems to be more intuitive and user-friendly?
How can we improve user satisfaction and efficiency when interacting with complex systems such as websites, mobile applications and software solutions?
How can we design digital work environments that promote well-being, productivity and creativity?
How can we consider the impact of new technologies such as artificial intelligence, automation and augmented reality on user behaviour and experience?
How can we better understand the different needs and expectations of users from different backgrounds, cultures and age groups and incorporate them into design processes?
How can we develop effective communication strategies and user interfaces in environments with high workloads and stress?
How can we design training and learning programs that are efficient, engaging and user-centric?
How can we evaluate and measure the effectiveness of measures and technologies to improve user experience and usability?
In short, UX and usability is about how users interact with digital products and systems and how these interactions can be optimised.
Why is UX and usability important?
These questions address real-world challenges and have a significant impact on improving user satisfaction, productivity and efficiency in the digital world. Research in UX and usability can lead to innovative solutions that benefit not only users, but also companies and society as a whole.
UX and usability lab: collecting and analysing user data
In order to answer these complex questions, various data modalities are collected and analysed. These include physiological signals, eye-tracking data, mouse movements, click behaviour and qualitative feedback from users.
By collecting and integrating these different data sources, both conscious and unconscious user reactions can be better understood.
Typical modalities that may be of interest in UX studies include:
Videos that record user interaction with the product
Screen recordings that are recorded time-synchronised with other cameras
User actions such as keystrokes and mouse clicks
Behavioural observations by human evaluators
Physiological signals such as heart rate or skin conductivity to record emotional reactions
Gaze data from eye trackers to analyse visual attention
Data from other programmes that provide temporal information
The synchronised integration of this data is crucial for a comprehensive understanding of the user experience in its diverse contexts.
Tools for UX Research: Mangold Observation Studio
Mangold Observation Studio is an innovative software for multimodal data collection and analysis in UX research.
With Mangold Observation Studio, you can conduct UX studies efficiently and effectively by recording various data such as video, physiology, screen activities and user interactions in a time-synchronised manner.
The live view allows all stakeholders, such as UX designers, researchers, product managers or developers, to follow the data in real time.
Mangold Observation Studio offers a user-friendly ‘one-push-button’ solution with extensive functions to improve UX research.
Successful Products through Professional Usability Studies
Successful companies have one thing in common: their products are highly valued by customers. And why? Most often, the key factor is user-friendliness, in other words, usability.
Whether it is websites, software programs, smartphones or machines – when products and services are developed it should always be about the people who use them.
Instead many a developer of machines or programs get lost in their love of technology and perform function overkill
But: most products don’t even use their full potential. They play a subordinate role and aren’t even the decisive factor for success on the market.
The key factor and unique selling point in competition is rather the intuitive handling – the usability.
Thus user- friendly is a decisive factor for a product’s success. To reach this added value, the development and implementation of a product should be the primary aim.
Because: User-friendly products can be the reason for a winning difference, plus these products are gaining more importance. The reason being, that users weigh the importance of usability as a purchase criteria. The same applies to the end-user, as well as for B2B relationships. Suppliers who integrate usability in their products achieve a higher customer satisfaction and for that reason are especially successful in the market.
User-friendly products reduce the learning time, minimize operating errors, increase the work production and escalate the working process.
A high usability supports the user in completing his tasks quickly, accurately, and to his satisfaction which helps in making it possible to reach goals intuitively.
Contact us for more information
Let's find your ideal user experience and usability lab solution together
Mangold International has many years of experience and has developed specialised solutions for the construction and operation of usability and user experience labs.
A Mangold Usability Lab is the ideal solution for conducting observational studies. Use our solutions to look directly into the customer's perspective.
Users of our solutions can be, for example, universities, service companies, financial service providers, travel providers, game manufacturers, web portal operators, hotel and catering businesses or industrial companies.
There are no limits to the imagination of possible applications.
