Touchless control of heavy equipment using low-cost hand gesture recognition

Thumbnail Image
Khaleghi, Leyla
Artan, Unal
Etemad, Ali
Marshall, Joshua
Human-machine interaction using remote hand gestures is becoming increasingly prevalent across various industries. However, their potential application to heavy construction equipment is often overlooked. This paper presents a robust and inexpensive hand gesture recognition system that was implemented and tested on a robotic 1-tonne wheel loader. The system uses an RGB camera paired with a laptop to process, in real time, hand gestures to control the loader. We first design 4 unique gestures for controlling the loader and then collect 26000 images to train and test a neural network for hand gesture recognition. Our system uses robust landmark detection using an off-the-shelf system prior to gesture recognition. We successfully controlled the loader to excavate in a rock pile by using the proposed hand gesture recognition system.