A Tutorial Series for Software Developers, Data Scientists, and Data Center Managers
We wanted to find an emotion recognition model that used images to predict multiple negative and positive emotions. To this end, we gathered images from existing databases that corresponded to positive, neutral, and negative emotions.
Our specific goals were twofold:
- Gather a 7,000-image database with negative and positive emotion ratings for each image—we aimed to represent images that were either emotionally negative or positive. Specifically, we aimed to represent seven specific emotions (two negative and five positive), and decided that we would need 1,000 images per emotion to train our emotion recognition model.
- Use the dataset to train a machine learning model that can recognize emotions in a new image and assign a vector of emotion ratings.
Additionally, we needed valence ratings reflecting the (un)pleasantness of emotion, for each image, in order to train and test the emotion classification model.
Limitations of Existing Image Databases
There are several existing image databases (EIDs), as highlighted in Table 1. These EIDs have four critical limitations:
- EIDs are often protected with copyrights and restrictions on use.
- Several EIDs have a limited set of categories or situations for inducing emotion.
- Several EIDs are better at inducing specific negative, rather than positive, emotions.
- EIDs are limited with a relatively small number of images, which is appropriate for several research purposes, but ultimately not sufficient for training and testing models.
Table 1:Existing image databases with emotion-related ratings.
|Database||Restricted Use?||Number of Pictures||Normative Ratings|
|The DIsgust-RelaTed-Images* (DIRTI*)1||No||300||Arousal, valence, disgust, and fear|
|EmoMadrid*2||Yes (academic)||700||Arousal and valence|
|Emotional Picture Set* (EmoPicS*)3||Yes (academic)||378||Valence, complexity, and excitement|
|Geneva Affective PicturE Database* (GAPED*)4||No||730||Arousal, valence, external norm, and internal norm|
|Image Stimuli for Emotion Elicitation* (ISEE*)5||No||356||Arousal, valence, dominance, and likeness|
|International Affective Picture Systems* (IAPS*)6||Yes (academic)||1,196||Arousal, valence, and dominance|
|Military Affective Picture System* (MAPS*)7||Yes (academic)||240||Arousal, valence, and dominance|
|Nencki Affective Picture System* (NAPS*)8||Yes (academic)||1,356||Arousal and valence|
|Open Affective Standardized Image Set* (OASIS*)9||No||900||Arousal and valence|
|Open Library of Affective Food* (OLAF*)10||No||95||Arousal, dominance, food craving, and pleasure|
|The Set of Fear Inducing Pictures* (SFIP*)8||Yes (academic)||956||Arousal, valence, and fear|
Figure 1. Examples of the types of scenes, animals, face, and objects depicted in emotional and neutral pictures used in existing image databases.
Creating Our Image Dataset
In light of the shortcomings of existing picture databases used in emotion research, we sought to create our own image dataset. Again, we wanted to develop an extensive picture database of 7,000 images, with two negative and five positive emotion ratings for each image.
To conduct our image dataset search and begin to create our database, we followed two key steps.
First, we identified a suitable image repository from which to scrape our images. We would need an extremely large repository that would allow us to flexibly search for various situations that might correspond with specific emotional experiences. Moreover, the results of these searches would need to be large enough and diverse enough to yield 7,000 unique images expected to differentially relate to seven emotions. A final and critical consideration was that we needed a repository that would allow other scientists and developers to use their images in future projects.
Inspired by the Image Stimuli for Emotion Elicitation* (ISEE*), which began with 10,696 images taken from Flickr* before narrowing down to 356 reliable images for emotion elicitation, we used pictures from Flickr’s Creative Commons* library to create our image dataset. Flickr represents a community of over 13 billion images, with its Creative Commons library cataloging pictures that are legally shared with varying degrees of restriction. We only scraped images from Flickr’s Creative Commons library without copyright restriction because our hope is to make our database available for researchers to use in their future work. Unlike many existing databases, we do not want to limit the use of our image catalog to researchers with an academic affiliation. Before scraping our images, we first needed to determine a set of search terms to help us find representations for each emotion of interest.
Second, we took a data-driven approach to determine sets of 20 search terms for image scraping for each target emotion. We wanted images that made viewers feel a certain way when they looked at them. When looking at an image, the scene or situation depicted in the image likely reminds the viewer of a time that he or she was in a similar situation, which might then also recall the emotions felt during that experience. Thus, we needed search terms for each emotion that would allow us to optimize our Flickr image query and identify pictures that should elicit our target emotions. Research has shown how:
- Anxiety is associated with situations that involve uncertainty and the potential for something bad to happen.
- Sadness is often felt when something bad has actually occurred, and these situations involve loss. We feel sad because something we did not want has happened, and we do not feel that we can deal with this situation.
- Joy is a pleasant emotion that we feel when a goal that we want has been obtained. Joy not only feels good, but also involves more excitement or intensity than some other positive emotions, such as tranquility.
- Tranquility is associated with situations that are pleasant and that lack urgency, encouraging a sense of being in the moment and savoring.
- Determination is unique from other positive emotions in that it involves the perception that a situation is not how an individual wants it to be.
- Awe is also a special positive emotion in that it is sometimes felt in response to negative situations. Studies have shown how awe is felt in response to grandiosity and things that extend beyond ourselves, such as scenic views of the Grand Canyon, and space.
To find pictures that captured the essence of these seven emotions, we took the following data-driven approach to determine our search terms:
- First, we had 700 Amazon Mechanical Turk* workers list at least 10 words that correspond to a target emotion.
- Second, we asked a separate group of workers to vote on the words that are most representative of a target emotion.
- Finally, we took the top 20 words for each target emotion and used these word sets as our search terms.
- With a focus on seven emotions, our aim was to develop a large, 7,000-picture database with data collected from multiple labelers on emotion ratings for each image, and to then use machine learning to classify images based on these emotion ratings.
- We conducted a rigorous literature search of the existing databases used in emotion research and concluded that these databases were insufficient for our purposes. Thus, we decided to use Flickr as a robust source for creating our unique image set.
- To pull images from Flickr, we took a data-driven approach to identify a set of search terms for each emotion of interest. Using these search times, we collected 1,000 appropriate images per target emotion.
- For developers and researchers interested in creating databases of their own, we highly encourage following a similar procedure, and examining the appropriateness of using current databases before deciding to create a new database.
- The DIsgust-RelaTed-Images (DIRTI) database: Validation of a novel standardized set of disgust pictures, Haberkamp et al., 2017
- Emotional Pictures Database
- Emotional Picture Set (EmoPicS)
- , Wessa et al., 2010,
- The Geneva affective picture database (GAPED): a new 730-picture database focusing on valence and normative significance, Dan-Glauser et al., 2011,
- Development and Validation of the Image Stimuli for Emotion Elicitation (ISEE), Kim et al., 2015,
- International Affective Picture System (IAPS), Lang et al., 2008,
- Military Affective Picture System (MAPS): A new emotion-based stimuli set for assessing emotional processing in military populations, Goodman et al., 2016,
- The Nencki Affective Picture System (NAPS): Introduction to a novel, standardized, wide-range, high-quality, realistic picture database, Marchewka et al., 2014
- Introducing the Open Affective Standardized Image Set (OASIS), Kurdi et al., 2017
- Affective Pictures and the Open Library of Affective Foods (OLAF): Tools to Investigate Emotions toward Food in Adults, Miccoli et al., 2016
|Prev: Set Up a Portable Experimental Environment for Deep Learning with Docker*||Next: Image Data Collection|
Create Applications with Powerful AI Capabilities
The Anatomy of an AI Team
Select a Deep Learning Framework
Select an AI Computing Infrastructure
Augment AI with Human Intelligence Using Amazon Mechanical Turk*
Crowdsourcing Word Selection for Image Search
Data Annotation Techniques
Set Up a Portable Experimental Environment for Deep Learning with Docker*
Image Dataset Search
Image Data Collection
Image Data Exploration
Image Data Preprocessing and Augmentation
Overview of Convolutional Neural Networks for Image Classification
Modern Deep Neural Network Architectures for Image Classification
Emotion Recognition from an Images Baseline Model
Emotion Recognition from Images Model Tuning and Hyperparameters
Music Dataset Search
Music Data Collection and Exploration
Emotion-Based Music Transformation
Deep Learning for Music Generation: Choosing a Model and Preprocessing
Deep Learning for Music Generation: Implementing the Model
TensorFlow Serving for AI API and Web App Deployment
Product and Performance Information
Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.