Hands-On AI Part 11: Image Dataset Search

Published: 10/12/2017   Last Updated: 10/12/2017

A Tutorial Series for Software Developers, Data Scientists, and Data Center Managers

We wanted to find an emotion recognition model that used images to predict multiple negative and positive emotions. To this end, we gathered images from existing databases that corresponded to positive, neutral, and negative emotions.

Our specific goals were twofold:

  1. Gather a 7,000-image database with negative and positive emotion ratings for each image—we aimed to represent images that were either emotionally negative or positive. Specifically, we aimed to represent seven specific emotions (two negative and five positive), and decided that we would need 1,000 images per emotion to train our emotion recognition model.
  2. Use the dataset to train a machine learning model that can recognize emotions in a new image and assign a vector of emotion ratings.

Additionally, we needed valence ratings reflecting the (un)pleasantness of emotion, for each image, in order to train and test the emotion classification model.

Limitations of Existing Image Databases

There are several existing image databases (EIDs), as highlighted in Table 1. These EIDs have four critical limitations:

  1. EIDs are often protected with copyrights and restrictions on use.
  2. Several EIDs have a limited set of categories or situations for inducing emotion.
  3. Several EIDs are better at inducing specific negative, rather than positive, emotions.
  4. EIDs are limited with a relatively small number of images, which is appropriate for several research purposes, but ultimately not sufficient for training and testing models.

Table 1:Existing image databases with emotion-related ratings.

Database Restricted Use? Number of Pictures Normative Ratings
The DIsgust-RelaTed-Images* (DIRTI*)1 No 300 Arousal, valence, disgust, and fear
EmoMadrid*2 Yes (academic) 700 Arousal and valence
Emotional Picture Set* (EmoPicS*)3 Yes (academic) 378 Valence, complexity, and excitement
Geneva Affective PicturE Database* (GAPED*)4 No 730 Arousal, valence, external norm, and internal norm
Image Stimuli for Emotion Elicitation* (ISEE*)5 No 356 Arousal, valence, dominance, and likeness
International Affective Picture Systems* (IAPS*)6 Yes (academic) 1,196 Arousal, valence, and dominance
Military Affective Picture System* (MAPS*)7 Yes (academic) 240 Arousal, valence, and dominance
Nencki Affective Picture System* (NAPS*)8 Yes (academic) 1,356 Arousal and valence
Open Affective Standardized Image Set* (OASIS*)9 No 900 Arousal and valence
Open Library of Affective Food* (OLAF*)10 No 95 Arousal, dominance, food craving, and pleasure
The Set of Fear Inducing Pictures* (SFIP*)8 Yes (academic) 956 Arousal, valence, and fear

Examples of the types of scenes
Figure 1. Examples of the types of scenes, animals, face, and objects depicted in emotional and neutral pictures used in existing image databases.

Creating Our Image Dataset

In light of the shortcomings of existing picture databases used in emotion research, we sought to create our own image dataset. Again, we wanted to develop an extensive picture database of 7,000 images, with two negative and five positive emotion ratings for each image.

To conduct our image dataset search and begin to create our database, we followed two key steps.

First, we identified a suitable image repository from which to scrape our images. We would need an extremely large repository that would allow us to flexibly search for various situations that might correspond with specific emotional experiences. Moreover, the results of these searches would need to be large enough and diverse enough to yield 7,000 unique images expected to differentially relate to seven emotions. A final and critical consideration was that we needed a repository that would allow other scientists and developers to use their images in future projects.

Inspired by the Image Stimuli for Emotion Elicitation* (ISEE*), which began with 10,696 images taken from Flickr* before narrowing down to 356 reliable images for emotion elicitation, we used pictures from Flickr’s Creative Commons* library to create our image dataset. Flickr represents a community of over 13 billion images, with its Creative Commons library cataloging pictures that are legally shared with varying degrees of restriction. We only scraped images from Flickr’s Creative Commons library without copyright restriction because our hope is to make our database available for researchers to use in their future work. Unlike many existing databases, we do not want to limit the use of our image catalog to researchers with an academic affiliation. Before scraping our images, we first needed to determine a set of search terms to help us find representations for each emotion of interest.

Second, we took a data-driven approach to determine sets of 20 search terms for image scraping for each target emotion. We wanted images that made viewers feel a certain way when they looked at them. When looking at an image, the scene or situation depicted in the image likely reminds the viewer of a time that he or she was in a similar situation, which might then also recall the emotions felt during that experience. Thus, we needed search terms for each emotion that would allow us to optimize our Flickr image query and identify pictures that should elicit our target emotions. Research has shown how:

  • Anxiety is associated with situations that involve uncertainty and the potential for something bad to happen.
  • Sadness is often felt when something bad has actually occurred, and these situations involve loss. We feel sad because something we did not want has happened, and we do not feel that we can deal with this situation.
  • Joy is a pleasant emotion that we feel when a goal that we want has been obtained. Joy not only feels good, but also involves more excitement or intensity than some other positive emotions, such as tranquility.
  • Tranquility is associated with situations that are pleasant and that lack urgency, encouraging a sense of being in the moment and savoring.
  • Determination is unique from other positive emotions in that it involves the perception that a situation is not how an individual wants it to be.
  • Awe is also a special positive emotion in that it is sometimes felt in response to negative situations. Studies have shown how awe is felt in response to grandiosity and things that extend beyond ourselves, such as scenic views of the Grand Canyon, and space.

To find pictures that captured the essence of these seven emotions, we took the following data-driven approach to determine our search terms:

  • First, we had 700 Amazon Mechanical Turk* workers list at least 10 words that correspond to a target emotion.
  • Second, we asked a separate group of workers to vote on the words that are most representative of a target emotion.
  • Finally, we took the top 20 words for each target emotion and used these word sets as our search terms.

Conclusion

  • With a focus on seven emotions, our aim was to develop a large, 7,000-picture database with data collected from multiple labelers on emotion ratings for each image, and to then use machine learning to classify images based on these emotion ratings.
  • We conducted a rigorous literature search of the existing databases used in emotion research and concluded that these databases were insufficient for our purposes. Thus, we decided to use Flickr as a robust source for creating our unique image set.
  • To pull images from Flickr, we took a data-driven approach to identify a set of search terms for each emotion of interest. Using these search times, we collected 1,000 appropriate images per target emotion.
  • For developers and researchers interested in creating databases of their own, we highly encourage following a similar procedure, and examining the appropriateness of using current databases before deciding to create a new database.

References

  1. The DIsgust-RelaTed-Images (DIRTI) database: Validation of a novel standardized set of disgust pictures, Haberkamp et al., 2017
  2. Emotional Pictures Database
  3. Emotional Picture Set (EmoPicS)
  4. , Wessa et al., 2010,
  5. The Geneva affective picture database (GAPED): a new 730-picture database focusing on valence and normative significance, Dan-Glauser et al., 2011,
  6. Development and Validation of the Image Stimuli for Emotion Elicitation (ISEE), Kim et al., 2015,
  7. International Affective Picture System (IAPS), Lang et al., 2008,
  8. Military Affective Picture System (MAPS): A new emotion-based stimuli set for assessing emotional processing in military populations, Goodman et al., 2016,
  9. The Nencki Affective Picture System (NAPS): Introduction to a novel, standardized, wide-range, high-quality, realistic picture database, Marchewka et al., 2014
  10. Introducing the Open Affective Standardized Image Set (OASIS), Kurdi et al., 2017
  11. Affective Pictures and the Open Library of Affective Foods (OLAF): Tools to Investigate Emotions toward Food in Adults, Miccoli et al., 2016


Prev: Set Up a Portable Experimental Environment for Deep Learning with Docker* Next: Image Data Collection

View All Tutorials ›

Product and Performance Information

1

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.