Abstract Detail

Paleobotany

Vaishnav, Mohit [1], FEL, Thomas [2], Rose, Jacob A [3], F Rodriguez Rodriguez, Ivan Felipe [4], Wilf, Peter [5], Serre, Thomas [6].

Visualizing how deep neural networks categorize living and fossil leaves.

Deep learning is starting to revolutionize science and society, and it is poised to transform botany. Because computers can effortlessly sift through millions of images, machine learning offers the promise to find visual features that are previously unknown or invisible to the human eye.   At the same time, interpreting how deep neural networks arrive at their categorization decisions remains a major challenge for computer vision. We have developed a deep-learning-based computer-vision system for identifying extant-leaf images to botanical family, using a new image database of cleared, x-rayed, and fossil leaves consisting mostly of angiosperms (see related abstracts at this meeting). The approach outperforms a previous system developed by our group (Wilf et al 2016, PNAS) and is arguably capable of discriminating among families beyond human capabilities. We describe novel machine-learning methods and visualizations for explaining the network's decisions to identify individual leaf images at the family level, a standard target for leaf and fossil-leaf identification. We leverage so-called explainability methods (Selvaraju et al 2017, ICCV) to produce a variety of heatmap visualizations that highlight important image locations the network uses to make classification decisions. In addition, we use modern optimization methods to synthesize images that are prototypical of each family to illustrate the most diagnostic visual features used by the system.   Finally, we implement a method based on the so-called concept activation vector (Kim et al 2018, ICML) to characterize the degree to which the system's strategy aligns with traits used by botanists (e.g., venation pattern, teeth, margin, shape). Overall, this work constitutes an initial step towards systematically investigating how large-scale machine methods can be used practically, for the benefit of human observers identifying living and fossil leaves.

1 - Universite de Toulouse, Artificial and Natural Intelligence Toulouse Institute, Toulouse, France
2 - Universite de Toulouse, Artificial and Natural Intelligence Toulouse Institute, France
3 - Brown University, School of Engineering & Cognitive, Linguistics and Psychological sciences, 184 Hope St Providence, Providence, RI, 02912, USA
4 - Brown University , Cognitive, Linguistic & Psychological Sciences, 184 Hope St Providence, RI, 02912, Providence, RI, 02912, USA
5 - Pennsylvania State University, Dept. of Geosciences, University Park, PA, 16802, USA
6 - Brown University , Cognitive, Linguistic & Psychological Sciences , 184 Hope St Providence, Providence, RI, 02912, USA

Keywords:
Machine learning
Paleobotany
Explainability
Dataset.

Presentation Type: Oral Paper
Session: PL2, Paleobotany: Cookson Student Presentations - Session II
Location: /
Date: Monday, July 19th, 2021
Time: 1:45 PM(EDT)
Number: PL2006
Abstract ID:318
Candidate for Awards:Isabel Cookson Award