Define the target before training
A harvesting system needs an operational definition of eligibility. Colour may help estimate maturity, but it is not a universal measure of eating quality or storage readiness. Variety, destination market and grower practice all matter. A tomato intended for transport may be selected differently from a strawberry sold nearby. The model should support that specification rather than silently replacing it with whatever was easiest to label.
Write separate labels for visible crop, apparently mature crop, damaged crop and accessible crop. Otherwise, a detector can look accurate while proposing targets that the machine cannot harvest. An orchard image may contain an obvious apple whose stem is hidden behind a branch. In a greenhouse, a pepper may be visible while its approach corridor intersects a support string. Detection and actionability should remain distinct in both the interface and the trial record.
A location is more than a pixel
An image detector usually identifies a region or outline. The arm needs a position in physical space and often an orientation. Stereo cameras, structured light and time-of-flight sensors can estimate depth, but each has limits. Bright light, glossy surfaces, thin stems and overlapping leaves can reduce the usefulness of a depth map. Missing depth should not be treated as free space.
The transformation between camera coordinates and robot coordinates also matters. Calibration can drift after vibration, maintenance or an impact. A system that places a marker accurately on screen may still approach the wrong location with its tool. Evaluate the entire mapping from observation to contact, and check calibration under the load and mounting arrangement used in the field.
Test the conditions that change the answer
Sun angle can change within a working shift. Leaves may become wet, fruit can move in wind and the sensor may collect dust. Training images from a single neat row do not establish performance across these conditions. A useful evaluation records environment, variety, canopy condition and camera settings so that failures can be traced to an operating envelope rather than dismissed as exceptions.
Lighting control can help, particularly in protected crops, but it introduces its own design questions. Ask whether the published result depends on a shroud, active illumination, a particular time of day or manual leaf removal. These measures may be perfectly reasonable. They simply need to be included in the cost, access requirements and labour assumptions of the proposed operation.
| Condition | Likely uncertainty | Useful check |
|---|---|---|
| Overlapping leaves | Incomplete crop shape | Compare a second viewpoint |
| Reflective fruit | Unreliable depth | Inspect depth validity at contact |
| Changing varieties | Shifted maturity cues | Validate labels with the grower |
Make uncertainty an operating state
A capable system should be allowed to abstain. If the crop is uncertain, it may obtain another view, defer the target or request help. Forcing every detection into a pick attempt can increase damage and consume time in repeated failures. Equally, excessive caution can leave valuable produce behind. Both outcomes belong in the performance report.
Confidence scores from a neural network are not automatically calibrated probabilities of a successful harvest. A high score may describe resemblance to training examples, not safe clearance or correct maturity. Ask how confidence thresholds were chosen and whether they were tested on rows, dates and growing sites excluded from training. Splitting adjacent photographs randomly can make an evaluation easier than deployment.
Failure mode: the fruit moved after detection
Wind or arm contact can move the target between observation and approach. A recovery strategy might refresh the estimate near contact or stop when geometry becomes inconsistent. Continuing along an obsolete trajectory is not a satisfactory default.
Request evidence that matches the decision
Precision and recall are useful for detection, but they do not describe all downstream losses. Request a target audit that distinguishes missed eligible crops, incorrect maturity choices, localisation errors and targets abandoned because access was blocked. A second record should show which proposed targets became accepted produce. This keeps perception failures separate from handling failures without pretending the two are independent.
Research by Sa and colleagues on deep learning for fruit detection illustrates why recognition deserves its own evaluation. Reviews by Bac and colleagues place perception within the wider harvesting problem. Neither kind of evidence should be converted directly into a claim about unattended daily output. For a purchase decision, insist on a demonstration using your crop specification and an agreed record of skipped targets.
Sources and further reading
- Sa et al., “DeepFruits: A Fruit Detection System Using Deep Neural Networks,” Sensors, 2016. Read the paper. Scope: image-based fruit detection, not a complete deployment guarantee.
- Bac et al., “Harvesting Robots for High-value Crops: State-of-the-art Review and Challenges Ahead,” 2014. Read the review.
- Wageningen University & Research and Harper Adams University provide wider agricultural robotics research context.