Figure-ground representation in deep neural networks

Brian Hu, Salman Khan, Ernst Niebur, Bryan Tripp

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Deep neural networks achieve state-of-the-art performance on many image segmentation tasks. However, the nature of the learned representations used by these networks is unclear. Biological brains solve this task very efficiently and seemingly effortlessly. Neurophysiological recordings have begun to elucidate the underlying neural mechanisms of image segmentation. In particular, it has been proposed that border ownership selectivity (BOS) is the first step in this process in the brain. BOS is a property of an orientation selective neuron to differentially respond to an object contour dependent on the location of the foreground object (figure). We explored whether deep neural networks use representations close to those of biological brains, in particular whether they explicitly represent BOS. We therefore developed a suite of in-silico experiments to test for BOS, similar to experiments that have been used to probe primate BOS. We tested two deep neural networks trained for scene segmentation tasks (DOC [1] and Mask R-CNN [2]), as well as one network trained for object recognition (ResNet-50 [3]). Units in ResNet50 predominantly showed contrast tuning. Units in Mask R-CNN responded weakly to the test stimuli. In the DOC network, we found that units in earlier layers of the network showed stronger contrast tuning, while units in deeper layers of the network showed increasing BOS. In primate brains, contrast tuning seems wide-spread in extrastriate areas while BOS is most common in intermediate area V2 where the prevalence of BOS neurons exceeds that of earlier (V1) and later (V4) areas. We also found that the DOC network, which was trained on natural images, did not generalize well to the simple stimuli typically used in experiments. This differs from findings in biological brains where responses to simple stimuli are stronger than to complex natural scenes. Our methods are general and can also be applied to other deep neural networks and tasks.

Original languageEnglish (US)
Title of host publication2019 53rd Annual Conference on Information Sciences and Systems, CISS 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781728111513
DOIs
StatePublished - Apr 16 2019
Event53rd Annual Conference on Information Sciences and Systems, CISS 2019 - Baltimore, United States
Duration: Mar 20 2019Mar 22 2019

Publication series

Name2019 53rd Annual Conference on Information Sciences and Systems, CISS 2019

Conference

Conference53rd Annual Conference on Information Sciences and Systems, CISS 2019
CountryUnited States
CityBaltimore
Period3/20/193/22/19

Fingerprint

Brain
Tuning
Image segmentation
Neurons
Masks
Experiments
Object recognition
Deep neural networks
Primates

ASJC Scopus subject areas

  • Information Systems

Cite this

Hu, B., Khan, S., Niebur, E., & Tripp, B. (2019). Figure-ground representation in deep neural networks. In 2019 53rd Annual Conference on Information Sciences and Systems, CISS 2019 [8693039] (2019 53rd Annual Conference on Information Sciences and Systems, CISS 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CISS.2019.8693039

Figure-ground representation in deep neural networks. / Hu, Brian; Khan, Salman; Niebur, Ernst; Tripp, Bryan.

2019 53rd Annual Conference on Information Sciences and Systems, CISS 2019. Institute of Electrical and Electronics Engineers Inc., 2019. 8693039 (2019 53rd Annual Conference on Information Sciences and Systems, CISS 2019).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Hu, B, Khan, S, Niebur, E & Tripp, B 2019, Figure-ground representation in deep neural networks. in 2019 53rd Annual Conference on Information Sciences and Systems, CISS 2019., 8693039, 2019 53rd Annual Conference on Information Sciences and Systems, CISS 2019, Institute of Electrical and Electronics Engineers Inc., 53rd Annual Conference on Information Sciences and Systems, CISS 2019, Baltimore, United States, 3/20/19. https://doi.org/10.1109/CISS.2019.8693039
Hu B, Khan S, Niebur E, Tripp B. Figure-ground representation in deep neural networks. In 2019 53rd Annual Conference on Information Sciences and Systems, CISS 2019. Institute of Electrical and Electronics Engineers Inc. 2019. 8693039. (2019 53rd Annual Conference on Information Sciences and Systems, CISS 2019). https://doi.org/10.1109/CISS.2019.8693039
Hu, Brian ; Khan, Salman ; Niebur, Ernst ; Tripp, Bryan. / Figure-ground representation in deep neural networks. 2019 53rd Annual Conference on Information Sciences and Systems, CISS 2019. Institute of Electrical and Electronics Engineers Inc., 2019. (2019 53rd Annual Conference on Information Sciences and Systems, CISS 2019).
@inproceedings{e6611d4f0e3b496f9a0efcf8ff5311a7,
title = "Figure-ground representation in deep neural networks",
abstract = "Deep neural networks achieve state-of-the-art performance on many image segmentation tasks. However, the nature of the learned representations used by these networks is unclear. Biological brains solve this task very efficiently and seemingly effortlessly. Neurophysiological recordings have begun to elucidate the underlying neural mechanisms of image segmentation. In particular, it has been proposed that border ownership selectivity (BOS) is the first step in this process in the brain. BOS is a property of an orientation selective neuron to differentially respond to an object contour dependent on the location of the foreground object (figure). We explored whether deep neural networks use representations close to those of biological brains, in particular whether they explicitly represent BOS. We therefore developed a suite of in-silico experiments to test for BOS, similar to experiments that have been used to probe primate BOS. We tested two deep neural networks trained for scene segmentation tasks (DOC [1] and Mask R-CNN [2]), as well as one network trained for object recognition (ResNet-50 [3]). Units in ResNet50 predominantly showed contrast tuning. Units in Mask R-CNN responded weakly to the test stimuli. In the DOC network, we found that units in earlier layers of the network showed stronger contrast tuning, while units in deeper layers of the network showed increasing BOS. In primate brains, contrast tuning seems wide-spread in extrastriate areas while BOS is most common in intermediate area V2 where the prevalence of BOS neurons exceeds that of earlier (V1) and later (V4) areas. We also found that the DOC network, which was trained on natural images, did not generalize well to the simple stimuli typically used in experiments. This differs from findings in biological brains where responses to simple stimuli are stronger than to complex natural scenes. Our methods are general and can also be applied to other deep neural networks and tasks.",
author = "Brian Hu and Salman Khan and Ernst Niebur and Bryan Tripp",
year = "2019",
month = "4",
day = "16",
doi = "10.1109/CISS.2019.8693039",
language = "English (US)",
series = "2019 53rd Annual Conference on Information Sciences and Systems, CISS 2019",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
booktitle = "2019 53rd Annual Conference on Information Sciences and Systems, CISS 2019",

}

TY - GEN

T1 - Figure-ground representation in deep neural networks

AU - Hu, Brian

AU - Khan, Salman

AU - Niebur, Ernst

AU - Tripp, Bryan

PY - 2019/4/16

Y1 - 2019/4/16

N2 - Deep neural networks achieve state-of-the-art performance on many image segmentation tasks. However, the nature of the learned representations used by these networks is unclear. Biological brains solve this task very efficiently and seemingly effortlessly. Neurophysiological recordings have begun to elucidate the underlying neural mechanisms of image segmentation. In particular, it has been proposed that border ownership selectivity (BOS) is the first step in this process in the brain. BOS is a property of an orientation selective neuron to differentially respond to an object contour dependent on the location of the foreground object (figure). We explored whether deep neural networks use representations close to those of biological brains, in particular whether they explicitly represent BOS. We therefore developed a suite of in-silico experiments to test for BOS, similar to experiments that have been used to probe primate BOS. We tested two deep neural networks trained for scene segmentation tasks (DOC [1] and Mask R-CNN [2]), as well as one network trained for object recognition (ResNet-50 [3]). Units in ResNet50 predominantly showed contrast tuning. Units in Mask R-CNN responded weakly to the test stimuli. In the DOC network, we found that units in earlier layers of the network showed stronger contrast tuning, while units in deeper layers of the network showed increasing BOS. In primate brains, contrast tuning seems wide-spread in extrastriate areas while BOS is most common in intermediate area V2 where the prevalence of BOS neurons exceeds that of earlier (V1) and later (V4) areas. We also found that the DOC network, which was trained on natural images, did not generalize well to the simple stimuli typically used in experiments. This differs from findings in biological brains where responses to simple stimuli are stronger than to complex natural scenes. Our methods are general and can also be applied to other deep neural networks and tasks.

AB - Deep neural networks achieve state-of-the-art performance on many image segmentation tasks. However, the nature of the learned representations used by these networks is unclear. Biological brains solve this task very efficiently and seemingly effortlessly. Neurophysiological recordings have begun to elucidate the underlying neural mechanisms of image segmentation. In particular, it has been proposed that border ownership selectivity (BOS) is the first step in this process in the brain. BOS is a property of an orientation selective neuron to differentially respond to an object contour dependent on the location of the foreground object (figure). We explored whether deep neural networks use representations close to those of biological brains, in particular whether they explicitly represent BOS. We therefore developed a suite of in-silico experiments to test for BOS, similar to experiments that have been used to probe primate BOS. We tested two deep neural networks trained for scene segmentation tasks (DOC [1] and Mask R-CNN [2]), as well as one network trained for object recognition (ResNet-50 [3]). Units in ResNet50 predominantly showed contrast tuning. Units in Mask R-CNN responded weakly to the test stimuli. In the DOC network, we found that units in earlier layers of the network showed stronger contrast tuning, while units in deeper layers of the network showed increasing BOS. In primate brains, contrast tuning seems wide-spread in extrastriate areas while BOS is most common in intermediate area V2 where the prevalence of BOS neurons exceeds that of earlier (V1) and later (V4) areas. We also found that the DOC network, which was trained on natural images, did not generalize well to the simple stimuli typically used in experiments. This differs from findings in biological brains where responses to simple stimuli are stronger than to complex natural scenes. Our methods are general and can also be applied to other deep neural networks and tasks.

UR - http://www.scopus.com/inward/record.url?scp=85065210334&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85065210334&partnerID=8YFLogxK

U2 - 10.1109/CISS.2019.8693039

DO - 10.1109/CISS.2019.8693039

M3 - Conference contribution

T3 - 2019 53rd Annual Conference on Information Sciences and Systems, CISS 2019

BT - 2019 53rd Annual Conference on Information Sciences and Systems, CISS 2019

PB - Institute of Electrical and Electronics Engineers Inc.

ER -