Meeting Abstract

P1-3   -   Synthetic datasets based on photogrammetry models power robust deep neural networks for behavioural analyses in social insects Plum, F*; Bulla, R; Imirzian, N; Labonte, D; Department of Bioengineering, Imperial College London, London, UK; The Pocket Dimension, Munich, Germany; Department of Bioengineering, Imperial College London, London, UK; Department of Bioengineering, Imperial College London, London, UK fabian.plum18@imperial.ac.uk http://evo-biomech.ic.ac.uk/

State-of-the-art machine learning methods promise to transform behavioural research by allowing researchers to gather data at an unprecedented scale. Applying transfer learning strategies to Deep Neural Networks (DNNs) – i.e., pre-training DNNs on highly variable datasets such as ImageNet – has made it possible to achieve human-level performance with minimum hand-labelled training data under well-defined conditions. However, porting these methods to variable and uncontrollable field conditions still poses a major challenge, because their robustness and versatility is limited by the variability and quantity of the supplied training data. Here, we aim to overcome this limitation by training DNNs on extensive synthetic datasets based on quasi-photorealistic computer-generated images. We developed an open-source photogrammetry and data generator pipeline which automatically creates annotated images containing an arbitrary number of insects, randomly posed, and rendered from digital 3D models, in procedurally generated environments. We demonstrate the potential of this approach by using leaf-cutter ants as a model system. Leaf-cutter ants are iconic herbivores that show an extensive, size-specific ‘division of labour’, i.e., colony workers of vastly different sizes preferentially perform different tasks. Studying the rules underlying this division of labour has been historically challenging due to the large number of individuals and the high frequency of partial or full occlusions. DNNs trained on our synthetic datasets are capable of accurate and remarkably robust detection, weight-, as well as pose-estimation, suggesting that this approach may enable us to build computer vision systems that can be deployed in realistic field scenarios with minimal labelling effort.