Hand gesture-based control of a front-end loader

Abstract

In this paper, we present the design and use of an instrumented glove consisting of a 9-DOF inertial measurement unit (IMU) and resistive flex sensors. The glove is used as a unique human-machine interface to control a Kubota R520s front-end loader, through input gestures, for the excavation of a fragmented rock pile. Raw sensor data from the glove is recorded and transmitted to a computer for gesture recognition. Recognized gestures are then used to command the loader to switch between dig states and control the excavation process. The system allows an operator to observe the entire process from beside the loader, providing them with valuable information about interactions between the loader bucket and rock pile not usually available when seated in the vehicle's cab. Preliminary experiments show that a novice operator was able to improve their performance by using the proposed system, evaluated based on metrics of total and dig completion times, as well as payload.