HYBRID EVENT: You can participate in person at Paris, France or Virtually from your home or work.
Agri 2022

Localisation and classification of broccoli heads

Cornelis Heemskerk

Cornelis Heemskerk, Speaker at Agriculture Conferences 2022
Inholland University of Applied Science, Netherlands
Title : Localisation and classification of broccoli heads


Fresh market broccoli is traditionally harvested selectively, in a labour-intensive, manual process. Only broccoli heads of the right size and right quality are harvested. The size assessment is done visually by the harvesters. As a first step trying to automate the selective broccoli harvesting process, the presentation describes how we have selected and used a depth camera and neural network to localise and measure the size of the broccoli heads.

The first step is to record in the field a series of images, annotate them and then to use a deep learning algorithm to detect broccoli heads in an image. Using a customized data set, based on the existing University of Lincoln broccoli dataset, we trained and compared the performance of two modern deep learning algorithms (YOLO V4 and YOLACT).

In a comparison between five different depth cameras, we selected the Intel Realsense stereo camera D435I as the most suitable for our purpose to perceive a broccoli in the field. After training on a custom data set, YOLO was found to have an Average Precision (AP) score of 70.48%, and a run-time performance of object detection within 32 ms. YOLACT was found to be better at accurately identifying and locating the broccoli head (AP score of 73.82%), however, YOLACT performs slower with a detection time of 51 ms.

A user interface was developed to display all detection information in real time. With the help of a class diagram, we will present an object-oriented overview of the proof-of-concept system and a sequence diagram to show how the objects work together.

With a subsequent field experiment, the recognition algorithm was tested on 50 new broccoli heads, which were scanned and measured to establish the ground truth in diameter and depth. During the field experiment it was shown that the diameter measurement is accurate to 9.36 mm and the depth measurement to 8.98 mm. In further work towards a machine that can selectively harvest broccoli, we will study how the vision system can be integrated with a harvesting arm to harvest broccoli in the field.



Dr. Heemskerk studied mechanical engineering at the Delft University of Technology (MSc 1985), and researched  assembly with industrial robots (PhD 1990). During 1985-1986 he was a visiting scientist at the Robotics Institute of Carnegie Mellon University, Pittsburgh, PA, USA.

Dr. Heemskerk is one of the main designers of the European Robotic Arm (ERA). The space robot was launched to ISS in 2021. In 2007, Dr. Heemskerk founded Heemskerk Innovative Technology, and has worked as robotics expert on many different projects.

Since 2016, dr. Heemskerk is a professor in Robotics at Inholland University of Applied Sciences, working on Smart Farming.

Signup for updates

By submitting this form, you are consenting to receive emails and notifications from Magnus Group. You can revoke your consent to receive emails at any time by using the Safe Unsubscribe link, found at the bottom of every email