Meeting Abstract

SICB+    An adaptable approach to multiple animal tracking in Python Peters, JM*; Jaramillo, J; Kirstin, KH; Cornell University; Cornell University; Cornell University jmp547@cornell.edu https://www.jacobmpeters.com

Many animal behavior studies require analysis of video data of multiple animals moving in space. This involves detecting individual animals, tracking their movement from frame to frame and resolving situations where multiple animals interact or overlap. Approaches to multiple animal tracking range from manual annotation, marker-based tracking, blob detection and data association, optical flow algorithms and brute force detection using convolutional neural network (CNN) models. While many open source tracking softwares have been published, many of them are either too specialized to adapt to new applications or have graphical user interfaces (GUIs) that make them easy to use but hard to modify. As a result, it can be difficult for users to get existing tracking tools to work for their study systems and many lab groups end up developing their own when starting new projects. We developed a modular multiple animal tracking software in Python that is intended to be adaptable to new applications and to lower the barrier to entry. Our approach generates animal detections through hierarchical thresholding/erosion algorithm, validates these detections using a simple CNN model and associates validated detections from frame to frame using a motion model-informed Hungarian algorithm. In cases where a track fails to be associated with a new detection, we use the Lucas Kenade Optical Flow algorithm to track image features until a new detection is found. Detection, validation and tracking are performed by separate python classes (or modules) allowing users to customize or replace one class without affecting the others. We implement our tracking software to track hundreds of honeybees, HexBug robots and pairs of zebrafish. We also explore how the performance scales with group size and video resolution and provide recommendations for how to use our software and adapt it to future projects.