Automatic Attribute Discovery with Neural Activations

Automatic Attribute Discovery with Neural Activations

Sirion Vittayakorn University of North Carolina at Chapel Hill, USA    Takayuki Umeda NTT Media Intelligence Laboratories, Japan    Kazuhiko Murasaki NTT Media Intelligence Laboratories, Japan    Kyoko Sudo NTT Media Intelligence Laboratories, Japan    Takayuki Okatani Tohoku University, Japan    Kota Yamaguchi Tohoku University, Japan
Abstract

How can a machine learn to recognize visual attributes emerging out of online community without a definitive supervised dataset? This paper proposes an automatic approach to discover and analyze visual attributes from a noisy collection of image-text data on the Web. Our approach is based on the relationship between attributes and neural activations in the deep network. We characterize the visual property of the attribute word as a divergence within weakly-annotated set of images. We show that the neural activations are useful for discovering and learning a classifier that well agrees with human perception from the noisy real-world Web data. The empirical study suggests the layered structure of the deep neural networks also gives us insights into the perceptual depth of the given word. Finally, we demonstrate that we can utilize highly-activating neurons for finding semantically relevant regions.

Keywords:
Concept discovery, Attribute discovery, Saliency detection

1 Introduction

In a social photo sharing service such as Flickr, Pinterest or Instagram, a new word can emerge at any moment, and even the same word can change its semantics and transforms our vocabulary set at any time. For instance, the word wicked (literally means evil or morally wrong) is often used as a synonym of really among teenagers in these recent years - “Wow, that game is wicked awesome!”. In such a dynamic environment, how can we discover emerging visual concepts and build a visual classifier for each concept without a concrete dataset? It is unrealistic to manually build a high-quality dataset for learning every visual concepts for every application domains, even if some of the difficulty can be mitigated by the human-in-the-loop approach [1, 2]. All we have are the observations but not definitions, provided in the form of co-occurring words and images.

In this paper, we consider an automatic approach to learn visual attributes from the open-world vocabulary on the Web. There have been numerous attempts of learning novel concepts from the Web in the past [3, 4, 5, 6, 7]. What distinguishes our work from the previous efforts is in that we try to understand potentially-attribute words in terms of perception inside deep neural networks. Deep networks have demonstrated outstanding performance in object recognition [8, 9, 10], and successfully applied to a wide range of tasks including learning from noisy data [11, 12] or sentiment analysis [13, 14]. In this paper, we focus on the analysis of neural activations to identify the degree of being visually perceptible, namely visualness of a given attribute, and take advantage of the layered structure of the deep model to determine the semantic depth of the attribute.

We collect two domain-specific datasets from online e-commerce and social networking websites. We study in domain-specific data rather than trying to learn general concept on the Web [3, 5] to isolate contextual dependency of attributes to object categories. For example, the term red eye can refer to an overnight airline flight or an eye that appears red due to illness or injury. This contextual dependency can cause an ambiguity for the visual classifier (red classifier); i.e., sense disambiguation. In this paper, we use a single-domain dataset to reduce such a semantic shift to study consistent meaning of attributes under the fixed context [15, 7].

Figure 1: Our attribute discovery framework.

We show that, using a trained neural network, we are able to characterize a visual attribute word by the divergence of neural activations in the weakly-annotated data. Figure 1 illustrates our framework. Our approach starts by cleaning the noisy Web data to find potentially visual attributes in the dataset, then splits the data into positive and negative sets. Using a pre-trained neural network, we identify highly activating neurons by KL divergence of activations. We show that we can use the identified neurons (prime units) for 1) learning a novel attribute classifier that is close to human perception, 2) understanding perceptual depth of the attribute, and 3) identifying attribute-specific saliency in the image. We summarize our contributions in the following.

  1. We propose to utilize the divergence of neural activation as a descriptor to characterize visual concept in the noisy weakly annotated dataset. The neurons identified by the divergence can help learn a visual attribute classifier that has a close proximity to human perception.

  2. We empirically study the relationship between human perception and the depth of activations to understand the visual semantics of attribute words.

  3. We show that the highly activating neurons according to the divergence are also useful for detecting attribute-specific saliency in the given image.

  4. We collect two noisy datasets from the Web to evaluate our framework. The empirical study shows we are able to learn a domain-specific visual attributes without manual annotation.

2 Related work

2.0.1 Attribute discovery

Our work is related to the recent work on concept discovery from a collection of images from the Web [3, 4, 5, 6, 16]. Early work by Ferrari et al.  [4] learns visual models of given attributes (e.g., red, spotted) from images collected from text search. NEIL [3] aims at discovering common sense knowledge from the Web starting from small exemplar images per concept. LEVAN [5] starts from mining bi-gram concepts from a large text corpus, and automatically retrieves training images from the Internet and learn a full-fledged detector for each concept. ConceptLearner [6] uses weakly labeled image collections from Flickr to train visual concept detectors. Shankar et al.  [17] study the attribute discovery in weakly-supervised scenario, where the goal is to identify co-occuring but missing attributes in the dataset while learning a deep network. Recent work by Sun et al.  [16] takes advantage of natural language to discover concepts for retrieval scenario. The automatic attribute discovery by Berg et al.  [7] is close to our work in that the work tries to evaluate visualness of the discovered synsets of attributes in the e-commerce scenario. The major difference of our approach from the previous works is that we aim at discovering attribute words and also characterizing the attribute perception using neural activations.

2.0.2 Neural representation

Thanks to the outstanding performance of deep neural networks in various tasks such as object recognition [8, 18, 9, 10] or domain adaptation for visual recognition task [19], the deep analysis of the intermediate representation of the neural networks has been getting more attention [20, 21, 22]. Escorcia et al.  [23] and Ozeki et al.  [24] study the relationship between neural representation and attributes. In this paper, we aim at utilizing the intermediate representation for visually characterizing unknown words in the noisy dataset, and study how the representation relates to human perception of attributes.

2.0.3 Class-specific saliency detection

Detecting class or attribute-specific saliency has been studied in the past in various forms, for example, as co-segmentation [25], parts [19, 26] or latent parts discovery [27], and weakly-supervised  [28, 29] or fully-supervised labeling [30, 31]. While Simonyan et al.  [32] uses gradient as a class-specific saliency, we demonstrate that the receptive field of neurons [22] can effectively identify the attribute-specific regions with our activation divergence. Our neuron-level saliency detection performs comparable to gradient-based approach [32], and can also reveal us insight into how learning changes the neurons’ response to visual stimuli.

3 Datasets and pre-processing

3.1 Etsy dataset

Etsy dataset is a collection of data from the online market of handcrafted products. Each product listing in Etsy contains an image, a title, a product description, and various metadata such as tags, category, or price. We initially crawl over 2.8 million product pages from etsy.com. Considering the trade-off between dataset size and domain specificity, we select the product images under the clothing category, which include 247 subcategories such as clothing/women/dress.

3.1.1 Near-duplicate removal

As common in any Web data, the raw data from Etsy contain a huge amount of near-duplicates. The major characteristics of Etsy data are the following: There are many shops, but the number of sold items per shop exhibits a long-tail. The same shop tends to sell similar items, e.g., the same black hoodie in the same background with a different logo patch, and in an extreme case, just a few words (proper nouns) are different in the product description. Our near-duplicate removal is primarily designed to prevent such proper nouns from building up a category. We observe that without the removal, we severely suffer from overfitting and end up with meaningless results.

Based on the above observation, we apply the following procedure to remove near-duplicates in Etsy: 1) Group product listings by shop. 2) Compute a bag-of-words from title and description except English stop words for each item within the group. 3) Compute the cosine distance between all pairs of products. 4) Apply agglomerative clustering by thresholding the pairwise cosine distance. 5) Randomly pick one product from each cluster. We apply the duplicate-removal for all shops in the dataset, and for each shop we merge any pairs of product having less than cosine distance into the same cluster. After the near-duplicate removal, we observed that roughly 40% of the products in Etsy were considered near-duplicates. We obtained 173,175 clothing products for our experiment.

3.1.2 Syntactic analysis

Given the title and description of the product in Etsy dataset, we apply syntactic analysis [33] and extract part-of-speech (POS) tags for each word. In this paper, we consider 250 most frequent adjectives (JJ, JJR, and JJS tags) as potential attribute words. Unless noted, we use the (50%, 25%, 25%) splits for train, test, and validation in the following experiments.

3.2 Wear dataset

We crawled a large collection of images from the social fashion sharing website, wear.jp. The website contains an image, associated shots from different views, list of items, blog text, tags, and other metadata. The images in Wear dataset are extremely noisy; Many users take a photo with a mobile device under uncontrolled lighting conditions and inconsistent photo composition, making it very challenging to apply any existing fashion recognition approach [34]. From the crawled data, we use the random subset of 212,129 images for our experiments.

3.2.1 Merging synonyms and translations

From Wear dataset, we select user-annotated tags for candidate words. The majority of tags from Wear dataset are written in Japanese (some in English), but there are also multiple synonyms treated as different tags including typos. We observe such synonyms and translations creating many duplicates. To mitigate this problem, we remove synonyms by translating all words to English, using Google Translate, and merge words that maps to the same English word. After translation, we pick up the most frequent 250 tags as a set of attribute candidates and use for our experiment. Note that machine-translation is not perfect and we manually fix translation errors in the selected tags.

4 Attribute discovery

Our attribute discovery framework starts by first splitting the weakly-annotated dataset into positive and negative sets, then computes Kullback-Leibler divergence (KL) for each activation unit in the deep neural network. We use the KL divergence to determine the important neurons for the given attributes. With these selected neurons, we can estimate the degree of visualness of attributes.

4.1 Divergence of neural activations

Although the image representation (neural activations) from the deep network captures numerous discriminative features in an image [21], each neuron in the network only sparsely responds to visual stimuli. We attempt to find neurons that highly respond to the visual pattern associated with a given attribute word. We propose to use the KL divergence of activations to identify these highly responding neurons or prime units for the given attribute.

Our framework starts by splitting the dataset into positive and negative sets according to the weak annotation (adjectives or tags in Sec 3). Positive or negative sets , are images with or without the candidate attribute-word . Note the noisy annotation contains both false-positive and false-negative samples. Using a pre-trained neural network, we compute the empirical distribution of neural activations from all of the units in the network. Let us represent the empirical distribution of the positive / negative set by and for each neuron . For convolutional layers, we max-pool the spatial dimension in all channels and compute histograms , since the maximum response is sufficient to identify unique units regardless of the location. Finally, we compute the symmetric KL divergence for each activation unit of the network:

(1)

where is the activation of the unit corresponding to histogram bins. The resulting KL divergence serves as an indicator to find prime units for the word . The intuition is that if the word is associated to specific visual stimuli, the activation pattern of the positive set should be different from the negative set and that should result in a larger KL divergence for visual attributes (e.g., red, white, floral, stripped) than less visual attributes (e.g., expensive or handmade). In other word, we should be able to identify the visual pattern associated to the given word by finding neurons with higher KL divergence.

4.2 Visualness

We follow the previous work [7] and define the visualness in terms of the balanced classification accuracy given the positive and negative sets:

(2)

where is a binary classification function. To eliminate the bias influence in the accuracy, we randomly subsample the positive and negative sets to obtain balanced examples (50%-50%). We use neural activations as a feature representation to build a classifier, and use the KL divergence as resampling and feature-selection criteria to identify important features for a given word .

4.2.1 Selecting and resampling by activations

The noisy positive and negative sets bring undesirable influence when evaluating the classification accuracy of the word (eq 2). Here, we propose to learn a visual classifier in two steps; we first learn an initial classifier based only on the activations from prime units, then we rank images based on the classification confidence. After that, we learn a stronger classifier from the confident samples using all of the activations in the network. More specifically, we first select 100 prime units according to the KL divergence (eq 1), and use the activations from these units as a feature (100 dimensions) to learn an initial classifier111Gaussian Naive Bayes also works in our setting, but a stronger classifier such as SVM with RBF kernel tends to overfit. using logistic regression [35] and identify the confident samples for the second classifier.

4.2.2 Learning attribute classifier

Once we learn the initial classifier, we rank images based on the confidence, resample the same number of images from both positive and negative sets according to the ranked order, and learn another attribute classifier using logistic regression from all of the activations (9,568 dimensions). Although more than 8,000 activations are from FC layers, the information gain is not necessarily proportional to the number of dimensions; FC layers tend to fire only a handful neurons, whereas convolutional layers after max-pooling give dense activations. Finally, the accuracy evaluation (eq. 2) on the balanced test set gives the visualness of the given word.

4.3 Human perception

To evaluate our approach, we collect human judgment of visualness using crowdsourcing, and compute the correlation between our visualness and human perception. Following the observation in [36] that it is harder for humans to provide the absolute visualness score of attribute than the relative score. Thus, we design a task on Amazon Mechanical Turk as follows; given a word, we provide two images to the annotators where one is from the positive set and the other is from the negative set. We ask annotators to pick an image that is more relevant to the given attribute, or if there is none, answer none. We pick the 100 most frequent words in Etsy dataset for evaluation purpose. For each word, we randomly pick 50 pairs of positive and negative images, and asked 5 annotators to complete one task. We define the human visualness of word as the ratio of positive annotator agreements:

(3)

where is an indicator function, is the number of positive votes for image pair , is the number of annotated images, and is a threshold. We set for 5 annotators in our experiment. Eq. 3 allows us to convert the relative comparison into agreement score, which is in absolute scale.

4.4 Experimental results

We use the Etsy dataset to evaluate our visualness.222Due to the translation issues, we were not able to get reliable human judgments in Wear dataset. Since neurons activate differently depending on the training data, we compare the following models:

  • Pre-trained: Reference CaffeNet model [8] implemented in [37], pre-trained on ImageNet 1000 categories.

  • Attribute-tuned: A CNN fine-tuned to directly predict the weakly-annotated words in the dataset, ignoring the noise. We replaced the soft-max layer in CaffeNet with a sigmoid to predict 250 words (Sec 3.1).

  • Category-tuned: A CNN fine-tuned to predict the 247 sub-categories of clothing using metadata in Etsy dataset, such as t-shirt, dress, etc.

We choose the basic AlexNet to evaluate how fine-tuning affects in our attribute discovery task, but we can also apply a different CNN architecture such as VGG [9] to do the same. The category-tuned model is to see the effect of domain transfer without overfitting to the target labels. We compare the following different visualness definitions against human.

  • CNN+random: Randomly subsample the same number of positive and negative images, learn a logistic regression from all of the neural activations (9,568 dimensions) in CNN, and use the testing accuracy to define the visualness. This is similar to the visualness prediction in the previous work [7] except that we use neural activations as a feature.

  • CNN+initial: Testing accuracy of the initial classifier trained on the most activating neurons or prime units.

  • CNN+resample: Testing accuracy of the attribute classifier trained on the resampled images according to the confidence of the initial classifier and learned from all of the neural activations, as described in Sec 4.2.2.

  • Attribute-tuned: Average precision of the direct prediction of the Attribute-tuned CNN in the balanced test set. We choose average precision instead of accuracy due to severe overfitting to our noisy training data.

  • Language prior: The n-gram frequency of adjective-noun modification for the given attribute-word from the Google Books N-grams [38]. We show the language prior as a reference to understand the scenario when we do not have access to visual data at all. The assumption is that for each of the object category in Etsy, visual modifier should co-occur more than non-visual words. We compute the prior using the sum of n-gram probability on attribute-category modification to 20 nouns in the Etsy clothing categories.

4.4.1 Quantitative evaluation

Method Feature dim. Pearson Spearman
Pre-trained+random (baseline) 9,568 0.737 0.637
Pre-trained+initial 100 0.760 0.663
Pre-trained+resample 9,568 0.799 0.717
Attribute-tuned 4,096 0.662 0.549
Attribute-tuned+random 9,568 0.716 0.565
Attribute-tuned+initial 100 0.716 0.603
Attribute-tuned+resample 9,568 0.782 0.721
Category-tuned+random 9,568 0.760 0.684
Category-tuned+initial 100 0.663 0.480
Category-tuned+resample 9,568 0.783 0.704
Language prior - 0.139 0.032
Table 1: Visualness correlation to human perception.

Table 1 summarizes the Pearson and Spearman correlation coefficients to human perception using different definitions of visualness together with the feature dimension. Note that achieving the highest accuracy in classification does not mean the best proximity to human perception in the noisy dataset. The results show that even though the initial classifiers learn from only 100 dimensional feature from prime units, they achieve the higher Spearman correlation to human perception than the random baselines with much larger feature. Moreover, resampling images by the initial classifier confidence improves the correlation to human perception over the random baseline in all models. These results confirm that feature-selection and resampling using the high-KL neurons help discovering visual attributes in the noisy dataset.

The result also suggests directly fine-tuning against the noisy annotation can harm the representational ability of neurons. We suspect that fine-tuning to a domain-specific data with possibly non-visual word leads to overfitting and suppresses neurons’ activity even if they are important in recognition. The pre-trained network gives the slightly higher Pearson correlation perhaps because the neurons are trained on wider range of visual stimuli in the ImageNet than in a domain-specific data like Etsy, and that somehow helps reproducing human perception. The low correlation from language prior indicates the difficulty of detecting visual attributes only from textual knowledge.

4.4.2 Qualitative evaluation

Table 2 lists the most and least visual attributes for selected methods. Note that the error in syntactic analysis incorrectly marked some nouns as adjective, such as url or flip (flip-flops) here. Generally, CNN-based methods result in a similar choice of the words. Language prior is picking very different vocabulary perhaps due to the lack of domain-specific knowledge in Google Books.

Figure 2 shows examples of the most or least confident images according to the pre-trained+resample model. From concrete concepts like orange to more abstract concepts elegant, we confirm that our automatic approach can learn various attributes only from the noisy dataset. Figure 3 shows examples of the most and least floral images from both positive and negative sets. The noise in the dataset introduces a lot of false-negatives (not mentioned but actually floral product) and false-positives (mentioned floral in text but not relevant to the product). Our automatically learned attribute classifiers can function as a dataset purifier in a noisy dataset.

Method Most visual Least visual
Human flip pink red floral blue url due last right additional
sleeve purple little black yellow sure free old possible cold
Pre-trained+resample flip pink red yellow green big great due much own
purple floral blue sexy elegant favorite new free different good
Attribute-tuned flip sexy green floral yellow right same own light happy
pink red purple lace loose best small different favorite free
Language prior top sleeve front matching waist organic lightweight classic gentle adjustable
bottom lace dry own right floral adorable url elastic super
Table 2: Most and least visual attributes discovered in Etsy dataset.
Figure 2: Examples of most and least predicted images for some of the attributes.
Figure 3: Most and least floral images. With our automatically learned classifier, we can discover false-negatives and false-positives in the dataset.

5 Understanding perceptual depth

In this section, we explore how each layer in the neural networks relates to attributes. It is well-known that neurons in a different layer activate to different types of visual pattern [21, 23]. We further attempt to understand what type of semantic concepts directly relate to neurons using the KL divergence.

We consider the activation with respect to the layer depth. We compute the relative magnitude of max-pooled KL divergence for layer :

(4)

We are able to identify the most salient words by ranking attribute vocabulary based on . In the following experiments, we use 7 layers in CaffeNet.

We use both Etsy and Wear dataset for finding salient words at each layer. Table 3 lists the most salient words for each layer of the pre-trained CNN in the two datasets. We can clearly see that more primitive visual concepts like color (e.g., orange, green) appear in the earlier stage of the CNN, and as we move down the network towards the output, we observe more complex visual concepts. We can observe the same trend from both Etsy and Wear datasets even though the two datasets are very different. Note that there are non-visual words in a general sense due to the dataset bias. For example, genuine in Etsy tends to appear in the context of genuine leather, and many appear in the context of many designs available for sweatshirt products. Such dataset bias results in higher divergence of neurons’ activity. One approach to deal with such context-dependency is probably to consider a phrase instead of a word.

norm1 norm2 conv3 conv4 pool5 fc6 fc7
orange green bright flattering lovely many sleeve
colorful red pink lovely elegant soft sole
vibrant yellow red vintage natural new acrylic
bright purple purple romantic beautiful upper cold
blue colorful green deep delicate sole flip
welcome blue lace waist recycled genuine newborn
exact vibrant yellow front chic friendly large
yellow ruffle sweet gentle formal sexy floral
red orange french formal decorative stretchy waist
specific only black delicate romantic great american
(a) Etsy dataset
norm1 norm2 conv3 conv4 pool5 fc6 fc7
blue
green
red-black
red
denim-on-denim
denim-shirt
pink
denim
yellow
leopard denim-jacket
pink
red
red-socks
red-black
champion
blue
white
shirt
i-am-clumsy
yellow border-striped-tops
border-stripes
dark-style
stripes
backpack
red
dark-n-dark
denim-shirt
navy
outdoor-style kids
bucket-hat
hat-n-glasses
black
sleeveless
american-casual
long-cardigan
white-n-white
stole
mom-style shorts
half-length
pants
denim
dotted
border-stripes
white-pants
border-tops
gingham-check
sandals
chester-coat white-skirt
flared-skirt
spring
upper
beret
shirt-dress
overalls
hair-band
loincloth-style
matched-pair long-skirt
suit-style
midi-skirt
gaucho-pants
handmade
straw-hat
white-n-white
white-coordinate
white-pants
white
(b) Wear dataset
Table 3: Most salient words for each CNN layer.

5.0.1 How fine-tuning affects perceptual depth

Fine-tuning has an influence on the magnitude of the layer-wise max-pooled KL in that 1) the pre-trained model activates almost equally across layers and 2) the category-tuned model induced larger divergence in mid-layer (conv4), while 3) the attribute-tuned model activates more in the last layer (fc7). Figure 5 shows the relative magnitude of average layerwise max-pooled KL: . The attribute-tuning causes a direct change in the last layer as expected, whereas the category-tuning brings a representational change in the mid-layers. The result suggests the domain-specific knowledge is encoded inside the mid-to-higher level representation, but there are domain-agnostic features in the earliest layers perhaps useful for recognizing primitive attributes such as color. Moreover, we also observe that the set of salient words per layer stay similar after fine-tuning in either cases; earlier layers activate more on primitive attributes, color or texture, and later layers activate more on abstract words.

Figure 4: Relative magnitude of average layer-wise maximum KL divergence.
Figure 5: Pearson correlation coefficients between human visualness and max KL divergence of each CNN layer.

5.0.2 How each layer relates to human perception

Finally, we evaluate how each layer relates to human perception, using the annotation from Sec 4.3. Figure 5 plots Pearson correlation of the layer-wise maximum KL divergence (eq 4) against human visualness. We show the correlation of pre-trained and attribute-tuned CNNs. The plot suggests the activation of mid-layers is closer to human visualness perception, but interestingly, the last fully-connected layers give negative correlation. We think that the last layers are more associated to abstract words that are not generally considered visual by humans, but they are contextually associated in a domain-specific data as in genuine leather case.

6 Saliency detection

6.0.1 Cumulative receptive fields

We consider saliency detection with respect to the given attribute based on the receptive field [22]. The main idea is to accumulate neurons’ response in the order of the largest KL divergence. Following [22], we first apply a sliding-window occluder to the given image, feed the occluded image to the CNN, and observe the difference in activation as a function of the occluder location for unit . We take the occluder patch at from the mean image of the dataset, at different scales. In our experiment, we use , , and occluder size with stride size 4 for the image input to the CNN. After getting the response map , we apply a Gaussian filter with the scale proportional to the occluder size, and average out multiple scale responses to produce a single response map. The resulting response map can have either positive or negative peaks to the input pattern, and we heuristically negate and invert the response map if the map has negative peaks. We normalize the response map within scale. Let us denote this normalized response map of unit by . We compute the final saliency map by accumulating units ordered and weighted by the KL divergence:

(5)

where . We accumulate units by the largest unit divergence up to .

6.0.2 Human annotation

We use Wear dataset for saliency evaluation, since the images in Etsy dataset are mostly a single object appearing in the center of the image frame and there is merely a localization need. Similarly to Sec 4.3, we collect human annotation on the salient region for evaluation purpose. For the randomly selected set of 10 positive images for the most frequent 50 tags in Wear dataset, we ask 3 workers to draw bounding boxes around the relevant region to the specified tag-word. We consider pixels having 2 or more annotator votes to be the ground-truth salient regions. We discard images not having any worker agreement in the evaluation.

6.0.3 Experimental results

(a) Mean AP
(b) Mean IoU
Figure 6: Saliency detection performance in terms of (a) mean average precision and (b) mean IoU of the attribute-tuned model over the heat-map threshold.

Figure 6 plots the average performance from all the tags in terms of mean average precision (mAP) for predicting pixel-wise binary labels, and mean intersection-over-union (IoU) of the attribute-tuned model. We compute IoU for the binarized saliency map at different threshold . The plots show the performance with respect to the number of accumulations , as well as the baseline performance of the smoothed gradient magnitude [32] of the attribute-tuned model. The performance improves as more neurons accumulate in the saliency map according to the divergence, and gives on par or slightly better performance against the baseline. Note that even the pre-trained model can already reach the baseline by this simple accumulation based on KL divergence, without any optimization towards saliency. We observe improvement in both pre-trained and attribute-tuned models, but the pre-trained model tends to require more neurons. We believe that fine-tuning makes each neuron activate more to a specific pattern while reducing activations on irrelevant patterns, and that results in the diminishing accumulation effect. The result also suggests that visual attributes are combinatorial visual stimuli rather than some visual pattern detectable only with a single neuron.

Figure 7: Results of detected salient regions for the given attribute. The rightmost column shows failure cases due to distracting contexts or visibility issues.
Figure 8: Accumulating receptive fields by the largest KL divergence. As we add more neurons, the saliency heat-map becomes finer.

Figure 8 shows the detection results by human annotation and our cumulative receptive field using the pre-trained and fine-tuned CNNs, when the accumulation size is 1, 8, and 64. Also, Figure 7 shows the results of human annotation and the pre-trained CNN with accumulation size . Our saliency detection method works remarkably well even without fine-tuning. As we accumulate more neurons, the response map tends to produce finer localization. Accumulation helps most of the cases, but we observe failure cases when there is a distractor co-occuring with the given attribute. For example in Figure 7, detecting shorts fails because legs always appear with shorts and we end up with legs detector instead of shorts (distractor issue). Moreover, our method tends to fail when the target attribute is associated to only small regions in the image (visibility issue).

7 Conclusion

We have shown that we are able to discover and analyze a new visual attribute from noisy Web data using neural activations. The key idea is the use of highly activating neurons in the network, identified by the divergence of activation distribution in the weakly annotated dataset. Empirical study using two real-world data gives us insights that our approach can automatically learn a visual attribute classifier that has a perceptual ability similar to humans, the depth in the network relates to the depth of attribute perception, and the neurons can detect salient regions in the given image. In the future, we wish to further study the relationship and similarity between discovered visual attributes and how the network architecture changes the neural perception in the hierarchical structure.

7.0.1 Acknowledgement

This work was partly supported by JSPS KAKENHI Grant Number 15H05919

References

  • [1] Biswas, A., Parikh, D.: Simultaneous active learning of classifiers & attributes via relative feedback. In: CVPR. (2013) 644–651
  • [2] Branson, S., Wah, C., Schroff, F., Babenko, B., Welinder, P., Perona, P., Belongie, S.: Visual recognition with humans in the loop. In: Computer Vision–ECCV 2010. Springer (2010) 438–451
  • [3] Chen, X., Shrivastava, A., Gupta, A.: Neil: Extracting visual knowledge from web data. In: ICCV. (Dec 2013) 1409–1416
  • [4] Ferrari, V., Zisserman, A.: Learning visual attributes. In: Advances in Neural Information Processing Systems. (2007) 433–440
  • [5] Divvala, S., Farhadi, A., Guestrin, C.: Learning everything about anything: Webly-supervised visual concept learning. In: CVPR. (2014)
  • [6] Zhou, B., Jagadeesh, V., Piramuthu, R.: Conceptlearner: Discovering visual concepts from weakly labeled image collections. In: CVPR. (June 2015)
  • [7] Berg, T.L., Berg, A.C., Shih, J.: Automatic attribute discovery and characterization from noisy web data. In: ECCV. Springer (2010) 663–676
  • [8] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS. (2012)
  • [9] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv (2014) 1–14
  • [10] He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. arXiv 7(3) (2015) 171–180
  • [11] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, Xiaogang Wang: Learning from massive noisy labeled data for image classification. CVPR (2015) 2691–2699
  • [12] Vo, P.D., Ginsca, A., Borgne, H.L., Popescu, A.: On Deep Representation Learning from Noisy Web Images. arXiv (2015)
  • [13] You, Q., Luo, J., Jin, H., Yang, J.: Robust Image Sentiment Analysis using Progressively Trained and Domain Transferred Deep Networks. AAAI (2015) 381–388
  • [14] Jou, B., Chen, T., Pappas, N., Redi, M., Topkara, M., Chang, S.F.: Visual Affect Around the World: A Large-scale Multilingual Visual Sentiment Ontology. ACM Multimedia (1) (2015)
  • [15] Lampert, C.H., Nickisch, H., Harmeling, S.: Learning to detect unseen object classes by between-class attribute transfer. CVPR (2009) 951–958
  • [16] Sun, C., Gan, C., Nevatia, R.: Automatic concept discovery from parallel text and visual corpora. In: ICCV. (2015) 2596–2604
  • [17] Shankar, S., Garg, V.K., Cipolla, R.: DEEP-CARVING: Discovering visual attributes by carving deep neural nets. In: CVPR. (2015) 3403–3412
  • [18] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: CVPR 2015. (2015)
  • [19] Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Learning and Transferring Mid-Level Image Representations using Convolutional Neural Networks. CVPR (2014) 1717–1724
  • [20] Yosinski, J., Clune, J., Nguyen, A., Fuchs, T., Lipson, H.: Understanding neural networks through deep visualization. In: Deep Learning Workshop, International Conference on Machine Learning (ICML). (2015)
  • [21] Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. ECCV 8689 (2014) 818–833
  • [22] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Object Detectors Emerge in Deep Scene CNNs. ICLR (2014)  12
  • [23] Escorcia, V., Niebles, J.C., Ghanem, B.: On the relationship between visual attributes and convolutional networks. CVPR (2015) 1256–1264
  • [24] Ozeki, M., Okatani, T.: Understanding convolutional neural networks in terms of category-level attributes. In: ACCV. Springer (2014) 362–375
  • [25] Chai, Y., Rahtu, E., Lempitsky, V., Van Gool, L., Zisserman, A.: TriCoS: A Tri-level Class-Discriminative Co-segmentation Method for Image Classification. ECCV 7572 (2012) 794–807
  • [26] Simon, M., Rodner, E.: Neural Activation Constellations: Unsupervised Part Model Discovery with Convolutional Networks. CVPR 21(1) (2015) 1–37
  • [27] Kiapour, M.H., Yamaguchi, K., Berg, A.C., Berg, T.L.: Hipster wars: Discovering elements of fashion styles. ECCV (2014) 472–488
  • [28] Guillaumin, M., Kuttel, D., Ferrari, V.: ImageNet Auto-Annotation with Segmentation Propagation. International Journal of Computer Vision 110(3) (2014) 328–348
  • [29] Zhou, B., Khosla, A., A., L., Oliva, A., Torralba, A.: Learning Deep Features for Discriminative Localization. CVPR (2016)
  • [30] Pan, H., Jiang, H.: A deep learning based fast image saliency detection algorithm. arXiv (2016)
  • [31] Pan, H., Wang, B., Jiang, H.: Deep learning for object saliency detection and image segmentation. arXiv (2015)
  • [32] Simonyan, K., Vedaldi, A., Zisserman, A.: Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. ICLR (2014) 1–
  • [33] de Marneffe, M.C., MacCartney, B., Manning, C.D.: Generating typed dependency parses from phrase structure parses. In: Proceedings of LREC. Volume 6. (2006) 449–454
  • [34] Yamaguchi, K., Kiapour, M.H., Ortiz, L.E., Berg, T.L.: Retrieving similar styles to parse clothing. TPAMI 37(5) (2015) 1028–1040
  • [35] Fan, R.E., Chang, K.W., Hsieh, C.J., Wang, X.R., Lin, C.J.: LIBLINEAR: A library for large linear classification. Journal of Machine Learning Research 9 (2008) 1871–1874
  • [36] Parikh, D., Grauman, K.: Relative attributes. In: ICCV. Number Iccv, IEEE (2011) 503–510
  • [37] Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: Convolutional architecture for fast feature embedding. arXiv (2014)
  • [38] Michel, J.B., Shen, Y.K., Aiden, A.P., Veres, A., Gray, M.K., Team, T.G.B., Pickett, J.P., Holberg, D., Clancy, D., Norvig, P., Orwant, J., Pinker, S., Nowak, M.A., Aiden, E.L.: Quantitative analysis of culture using millions of digitized books. Science (2010)
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
45312
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description