Agricultural robots offer the possibility of automated data collection with a suite of complementary sensing modalities, concurrently, from large numbers of plants, at many different locations, under widely ranging environmental conditions. Large amounts of such data can enhance our ability to calibrate regression models or train classification algorithms, in particular deep learning networks, which are increasingly being used in the agricultural domain and require large training data sets . Examples of this capability is the use of deep networks for flower and fruit detection in tree canopies, and the “See and Spray” system that uses deep learning to identify and kill weeds . Data from robots from different growers could be shared and aggregated too, although issues of data ownership and transmission over limited bandwidth need to be resolved. The creation of large, open-access benchmark data sets can accelerate progress in this area. Furthermore, sensors on robots can be calibrated regularly, something which is important for high-quality, reliable data. Other ways to reduce uncertainty is for robots to use complementary sensors to measure the same crop property of interest, and fuse measurements , or to measure from different viewpoints. For example, theoretical work shows that if a fruit can be detected in n independent images, the uncertainty in its position in the canopy decreases with n. Multiple sensing modalities can also help disambiguate between alternative interpretations of the data or discover multiple causes for them. New sensor technologies,big plastic pots such as Multi-spectral terrestrial laser scanning which measures target geometry and reflectance simultaneously at several wavelengths can also be utilized in the future by robots to assess crop health and structure simultaneously.

Another major challenge is to sense all plant parts necessary for the application at hand, given limitations in crop visibility. Complicated plant structures with mutually visually occluding parts make it difficult to acquire enough data to reliably and accurately assess crop properties , recover 3D canopy structure for plant phenotyping or detect and count flowers and fruits for yield prediction and harvesting, respectively. This is compounded by our desire/need for high-throughput sensing which restricts the amount of time available to ‘scan’ plants with sensors moving to multiple viewpoints. Robot teams can be used to distribute the sensing load and provide multiple independent views of the crops. For example, fruit visibility for citrus trees has been reported to lie in the range between 40% and 70% depending on the tree and viewpoint , but rose to 91% when combining visible fruit from multiple perspective images . A complementary approach is to utilize biology and horticultural practices such as tree training or leaf thinning, to simplify canopy structures and improve visibility. For example, when V-trellised apple trees were meticulously pruned and thinned to eliminate any occlusions for the remaining fruits, 100% visibility was achieved for a total of 193 apples in 54 images, and 78% at the tree bottom with an average of 92% was reported in . Another practical challenge relates to the large volume of data generated by sensors, and especially high-resolution imaging sensors. Fast and cheap storage of these data onboard their robotic carriers is challenging, as is wireless data transmission, when it is required. Application-specific data reduction can help ease this problem. The necessary compute power to process the data can also be very significant, especially if real-time sensor based operation is desired. It is often possible to collect field data in a first step, process the data off-line to create maps of the properties of interest , and apply appropriate inputs in a second step.

However, inaccuracies in vehicle positioning during steps one and two, combined with increased fuel and other operation costs and limited operational time windows often necessitate an “on-the-go” approach, where the robot measures crop properties and takes appropriate action on-line, in a single step. Examples include variable rate precision spraying, selective weeding, and fertilizer spreading. Again, teams of robots could be used to implement on-the go applications, where slower moving speeds are compensated by team size and operation over extended time windows.Interaction via mass delivery is performed primarily through deposition of chemical sprays and precision application of liquid or solid nutrients . Delivered energy can be radiative or mechanical, through actions such as impacting, shearing, cutting, pushing/pulling. In some cases the delivered energy results in removal of mass . Example applications include mechanical destruction of weeds, tree pruning, cane tying, flower/leaf/fruit removal for thinning or sampling, fruit and vegetable picking. Some applications involve delivery of both material and energy. Examples include blowing air to remove flowers for thinning, or bugs for pest management ; killing weeds with steam or sand blown in air streams or flame ; and robotic pollination, where a soft brush is used to apply pollen on flowers . Physical interaction with the crop environment includes tillage and soil sampling operations , and for some horticultural crops it may include using robotic actuation to carry plant or crop containers , manipulate canopy support structures  or irrigation infrastructure. In general, applications that require physical contact/manipulation with sensitive plant components and tissue that must not be damaged have not advanced as much as applications that rely on mass or energy delivery without contact. The main reasons are that robotic manipulation which is already hard in other domains can be even harder in agricultural applications, because it must be performed fast and carefully, because living tissues can be easily damaged. Manipulation for fruit picking have received a lot of attention because of the economic importance of the operation .

Fruits can be picked by cutting their stems with a cutting device; pulling; rotation/twisting; or combined pulling and twisting. Clearly, the more complicated the detachment motion is, the more time-consuming it will be, but in many cases a higher picking efficiency can be achieved because of fruit damage reduction during detachment. Fruit damage from bruises, scratches, cuts, or punctures results in decreased quality and shelf life. Thus, fruit harvesting manipulators must avoid excessive forces or pressure, inappropriate stem separation or accidental contact with other objects .Contact-based crop manipulation systems typically involve one or more robot arms, each equipped with an end-effector. Fruit harvesting is the biggest application domain , although manipulation systems have been used for operations such as de-leafing , taking leaf samples , stomping weeds , and measuring stalk strength . Arms are often custom designed and fabricated to match the task; commercial,growing berries in containers off-the-shelf robot arms are also used, especially when emphasis is given on prototyping. Various arm types have been used, including cartesian, SCARA, articulated, cylindrical, spherical and parallel/delta designs.That is, the position of the target is estimated in the robot frame using sensors and the actuator/arm moves to that position using position control. Closed-loop visual servoing has also been used to guide a weeding robot’s or fruit-picking robot’s end effector. End-effectors for fruit picking have received a lot of attention and all the main fruit detachment mechanisms have been tried . Mechanical design and compliance have also been used to reduce the effects of variability and uncertainty. For example, properly-sized vacuum grippers can pick/suck fruits of various sizes without having to center exactly the end-effector in front of the targeted fruit . Also, a large variety of grippers for soft, irregular objects like fruits and vegetables have been developed using approaches that include from air , contact and rheological change . Once a fruit is picked, it must be transported to a bin. Two main approaches have been developed for fruit conveyance. One is applicable only to suction grippers and spherical fruits, and uses a vacuum tube connected to the end-effector to transport the picked fruit to the bin . In this case there is no delay because of conveyance, as the arm can move to the next fruit without waiting. However, the vacuum tube system must be carefully designed so that fruits don’t get bruised during transport.

The other approach is to move the grasped fruit to some “home” location where it can be released to a conveyance system  or directly to the bin. This increases transport time, which may hurt throughput. Clearly, there are several design and engineering challenges involved with this step.Combining high throughput with very high efficiency is a major challenge for physical interaction with crops in a selective, targeted manner; examples of such selective interactions are killing weeds or picking fruits or vegetables. For example, reported fruit picking efficiency in literature for single-arm robots harvesting apple or citrus trees ranges between 50% to 84%; pick cycle time ranges from 3 to 14.3s . However, one worker on an orchard platform can easily maintain a picking speed of approximately 1 apple per 1.5 seconds with efficiency greater than 95% . Hence, replacing ten pickers with one machine would require building a 10-40 faster robotic harvester that picks gently enough to harvest 95% of the fruit successfully, without damage, and do so at a reasonable cost! Several factors render this combination challenging to achieve. Living tissues can be easily damaged and handling them typically requires slow, careful manipulation that avoids excessive forces or pressures. Biological variation introduces large variability in physical properties such as shape, size, mass, firmness of the targeted plants or plant components. This variability, coupled with uncertainty in the sensing system and limitations in the performance of control systems can affect negatively the accuracy, speed, success rate and effectiveness of the operation. Reduced accuracy can cause damage to the targeted part of the plant or nearby plant parts , or the entire plant . It may also cause reduced throughput due to misses and repeats, or reduced efficiency if no repeats are attempted. Visual servoing/guidance of robot actuators can reduce uncertainty and increase efficiency, but uneven illumination, shadows cast by branches and leaves, partial occlusions, and branches acting as obstacles present significant challenges in real-world conditions . Guiding the end-effector by combining inputs from multiple cameras is an approach that could be adapted to agricultural settings . Another possible direction is using deep reinforcement learning to learn visual servoing that is robust to visual variation, changes in viewing angle and appearance, and occlusions . Innovative end-effector design and control can also increase throughput and efficiency. If stem-cutting is used, challenges include detecting and cutting quickly and robustly from a large range of approaches, in the presence of touching fruits and twigs. If pulling is used, the force required to detach fruits depends on the type and maturity of the fruits, the approach angle of the end-effector, and on whether rotation is also used. Some fruits require concurrent, controlled, synchronized rotation and pulling to reduce skin/peel damage at the stem-fruit interface , a task that is complex and not easily modeled. Deep reinforcement learning for grasping is a possible approach to build sophisticated controllers for such tasks. Innovations in materials, design and control for soft robots could also be adapted to fruit picking and crop handling in general . Another important factor is limited accessibility of the targeted plants or their parts by robot end-effectors. Accessibility can be limited by plant structure, positioning, interference with neighboring plants or structures, and robot design. For example, in robotic weeding, weeds that are very close to a crop-plant’s stem and hidden under its canopy are not easily accessible by the end-effector without damaging the crop . In fruit harvesting, fruits in tree canopies that are positioned behind other fruits, branches or trellis wires also have limited accessibility by robotic harvesting arms. Accessibility can be improved by introducing dexterous, multi-d of actuation systems. However, control complexity can reduce throughput; the overall system cost will also be higher. Breeding and horticultural practices can also be utilized to improve accessibility. For example, tree cultivars with smaller and simpler canopies, training systems that impose simpler – planar – canopy geometrical structures along fruit thinning operations can contribute to higher fruit accessibility/reachability. To some extent, it is the availability of trellised planar architectures and precision fruit thinning which result in very high fruit visibility and reachability that have enabled robotic harvesting to emerge recently as a potentially cost effective approach to mechanical fruit harvesting at commercial scale. However, the cost and required labor demand for maintaining meticulously thinned and pruned trellised trees can be very high. Moreover, not all fruit trees can be trained in such narrow, planar systems.