Weed-removal system based on artificial vision and movement planning by A* and RRT techniques

. The recent exploration of autonomous robotics tasks in agro-industry has permitted the integration of theories of artificial vision and mobile robotics with tasks in precision agriculture. Artificial vision allows for the classification of weeds and crops from images of plantations. With 3D-image processing systems, the location of the weeds is determined, and then the movement of the tool responsible for eradication is proposed. This article presents the solution for finding weeds within a crop field using classifiers and the integration of a 3D-vision system that builds a point cloud featuring the plants to safeguard, the weeds and the free space using Zed technology. With this information, search techniques such as A* (A star) and RRT (Rapidly exploring Random Tree) are used to determine the trajectory that the weed-removal tool must follow. The last feature is an integral part of an XYZ-positioning system, and this is part of a mobile robot dedicated to precision agriculture tasks.


Introduction
Precision agriculture has emerged as a necessary element for the quality and efficiency in food generation, since the expectation of producing a sufficient quantity, compared with the population of the world, has not yet been fulfilled.With advancements in robotics and the integration of GPS (Global Positioning Systems), farmers have gained tools that help them rapidly know the positions of areas of interest in their crops, monitor data on crop growth, soil maintenance and environmental factors, and even send automated and preprogrammed machinery to execute tasks (Stafford, 2000).There are several articles (Zhang, Wang, & Wang, 2002;Lamb & Brown, 2001) that propose precision agriculture as a global idea that implies responsible management of natural resources, variation in plantings, soil care and the use of technologies in agricultural settings to react quickly and in a controlled manner to minimize the use of chemicals and maximize production.Precision agriculture has been shown to be a set of tools that help the farmer react more quickly, improve his product and obtain more economic benefits.Companies such as Bosch with BoniRob, Robotnik with Guardian, or Naïo with Oz are examples of an emerging industry that has consolidated solutions into the integration of robots in various agricultural tasks.Research with experimental platforms, such as the one presented by Auat Cheein (Auat Cheein & Carelli, 2013), allows for vehicles dedicated to the inspection of crops.Bergerman (Bergerman et al., 2015) created an automated vehicle to assist with fruit trees tasks, where by means of location detection, global positioning and detection of landmarks, the robot moves throughout the crop.
Colombian farmers face an imminent challenge to improve their processes, ensure greater production and produce higher-quality products.The experience of the GIDAM group (Universidad Militar Nueva Granada), with projects such as surveillance with aerial platforms from a flight-control perspective, development of tools for soil preparation and crop conservation, and in the detection of weeds (Pulido Rojas, Solaque Guzmán, & Velasco Toledo, 2017), shows progress in this direction.Starting from the development of a mobile platform dedicated to agricultural work-the Ceres AgroBot (Figure 1 equipped with cameras and actuators to remove weeds and apply nutrients and fungicides of natural origin)-and with the use of mission planning, monitoring of trajectories, and process automation, the GIDAM group is increasing the capabilities of the Ceres AgroBot.
Acta Scientiarum.Agronomy, v. 41, e42687, 2019 Weeds are plants that compete with the desired commercial crop and lower productivity.They do this by blocking irrigation canals and by competing for water, nutrients, space, and light; consequently, the quality and crop yield decrease.A robotics application that can discriminate in images between weeds and crops is a cost-effective alternative to allow selective treatment to focus on optimizing resources and preserving environment by identifying and removing only weed plants that are mixed in with desired crops.This problem can be solved using image-processing to select undesired plants and perform autonomous mechanical eradication on a mobile platform that moves through the field without affecting the desired plants using chemical products (this is the method of the Ceres AgroBot).
The task of removing weeds poses interesting challenges: i) Using autonomous and Geo-referenced vehicles -for ease of transportation of tools within the crop.
ii) Identification of the weed within the crop -artificial intelligence techniques such as 3D-image processing allow the identification of an object in Euclidean space from the parameterization of the image.Recent developments in the field of machine vision have led to a renewed interest in implementing weed recognition systems based on machine vision.Essentially, there are three main approaches for weed detection based on color, shape or texture analysis.With respect to color and shape features, previous research suggests a criterion for segmenting plants based on a vegetation index that emphasizes the "green" component of the source image.Two such indices are the Excess Green Index (Muangkasem, Thainimit, Keinprasit, & Isshiki, 2010) and the Normalized Difference Vegetation Index, which are used for weed classification by color and shape (Pérez, López, Benlloch, & Christensen, 2000) and quantify map vegetative cover (Wiles, 2011).An advantage of indices is that they may, as a side effect, perform well under different sunlight and background conditions.Color features can be complemented with shape features that describe their geometry; if weeds can be identified using shape, then they can be identified using area, perimeter, convexity and longest chord calculations (Shinde & Shukla, 2014).Ahmed et al. (Ahmed, Al-Mamun, Bari, Hossain, & Kwan, 2012) evaluated fourteen color, size and momentinvariant features to obtain an optimal combination that provides the highest classification rate; their result achieves more than 97% accuracy.
iii) Eradication without chemical intervention or repetitive tasks when manual eradication is performedavoiding the use of chemicals while automating weed removal is a major challenge.The theory of trajectory planning allows us to address the displacement of the acting tool for the removal of weeds.It is supported in classical approaches, such as those described by Latombe (Latombe, 1991).The planning of the movement for a manipulator was presented by Benevides (Benevides & Grassi, 2016), where he used basic RRT algorithms.The trajectories found were sometimes not directly applicable, forcing the use of smoothing theories such as the Bezier curve theory.
Based on the studies described above, there is potential for using texture features to discriminate weeds and vegetables, as applied to classifiers' designs, 3D vision for reconstruction of work space and path planning to be carried out in agricultural robotic applications.The main objective of the research presented in this paper was to provide an approximation for an autonomous robot working in agricultural applications; thus, the following were shown: 1) the development of a useful algorithm to discriminate weeds, using image filtering to extract color and area features, where a process is implemented to label each object in the scene and a classification based on area is proposed that includes sensitivity, specificity, and positive and negative predicted values in order to evaluate algorithm performance; 2) a 3D system used to make a point cloud in order to create a work space (weeds, vegetables and free space); and 3) the use of path planning based in A*, RRT and Bezier to determine a trajectory for the weed-removal tool.The remainder of this paper is organized as follows.The first part is the Introduction.Section II introduces the methodology and a description of the main steps for the proposed method, image acquisition, marker detection, trajectory-prediction method, and robots in agriculture, focusing on the description of the Ceres AgroBot platform.Section III describes the experiments, including their results (A * algorithm, RRT algorithm and the Bezier technique).Section IV concludes this work.
Prior to commencing the study, theoretical foundation and flow process were established as follows: first (Section 2.1), some robots used for precision agriculture are introduced.Next, section 2.2, an imageacquisition description is introduced.Subsection 2.3 presents the stereo system.Section 2.4 details A*, RRT and Bezier for trajectory determination.
Finally, section 2.5 describes the strategy for Ceres AgroBot weed removal.

Robots in agriculture -Ceres AgroBot
Several companies and research laboratories have been working on autonomous vehicles that have systems that assist in agriculture.Examples of this are the BoniRob, Guardian, or Oz platforms, and mobile devices that have been conditioned and presented by Auat Cheein and Bergerman (Figure 2).The systems dedicated to autonomous tasks within agriculture usually have locomotion systems, global positioning sensors, proximity sensors (ultrasound, bumpers and/or lasers), inertial sensors, vision systems, communication systems and information processing elements (computers, embedded systems, video cards).In the case of Ceres AgroBot, its locomotion relies on a differential configuration powered by 5 KW electric motors with a capacity of 200 kg of load on the platform and a vision system formed by a thermal imaging camera-Flir, a Zed stereo camera, and a multispectral camera, MicaSense.It has laser Hokuyo sensors to check the elements next to the platform.At the processing level, it has a quad-core embedded computer, embedded systems and an image processing card such as a Nvidia-Jetson.The robot also has a nutrientcompost, pesticide, and weed-removing tool, all within the context of a positioning system in the XYZ Cartesian plane for focused work.
Acta Scientiarum.Agronomy, v. 41, e42687, 2019 The weed removal system associated with Ceres AgroBot consists in its base of a Cartesian positioning system in 3D space, with the objective of reaching the weed and removing it.It has linear motors, located on the X-axis, with 1200 mm of stroke; 900 mm of stroke on the Y-axis; and 600 mm of stroke on the Z-axis.The locomotion system is differential, with the direction of the vectors as fixed forces, making it a nonholonomic system.The model of the robot's kinematics is described by (1).
where k1 and k2 are coefficients attributed to the distances of the acting elements (motors) and the center of gravity (RG).
The position of the vehicle allows localization within the crop, important data when making Simultaneous Location and Mapping (SLAM).Thus, the location of the end effector is described with respect to the center of gravity of the platform.In this way, the position of the tool responsible for weed removal is fully parameterized and known.

Weed detection vision system
The goal is to achieve a baseline method for developing a real-time weed detection system through binary classification when vegetation is detected; that is, to separate soil and plants, then to apply feature extraction for discriminating weeds.First, a green plant detection algorithm is implemented to remove soil from the image so that image information is reduced.The next steps of the algorithm focus only on vegetation, after which median filtering removes noise as "salt and pepper", with the advantage of preserving edges.Third, the previous output is converted to binary; at this point, small objects are removed in order to avoid outliers.Next, the pixels bordering this area are labeled, and thus all objects in the image are identified.Finally, the area is calculated for each object.With the values obtained, we set a threshold to differentiate weed from crop such that the method is a feature-extraction criterion based on size.
The digital images are captured in outdoor light conditions with perspective projection over the crop.Captures are made with 8 MP resolution, in RGB color space, and with JPG extensions.The main idea of crop image acquisition is to avoid lighting and sharpness problems so that color changes with respect to vegetation are reduced.In this way, the accuracy of the first step of the plant classification algorithm can increase, provided that the green color scheme over the objects is maintained.
A method for the segmentation of green plants and separation from the background is to use the green component of the RGB color model to obtain an image with only vegetation information.It is appropriate to subtract the green component from the original image because the module for the weed remover robot has a camera obscura and lamps in order to maintain uniform illumination.Afterwards, a median filtering is used for noise suppression in filtered images, preserving edges whereby the relevant image information is conserved and tends to produce regions of constant or nearly constant intensity.It is necessary to segment the image in order to locate plants, assigning a label to each pixel and highlighting the similarity of the features used for detection of plants.Here, color and area serve as descriptors for a threshold classifier.The results of the segmentation are shown in Figure 3a.Because feature extraction is based on area, it is appropriate to fill in the holes in the image.Thus, evaluation in the next step is enhanced, since compact objects are obtained.For this purpose, an algorithm based on morphological reconstruction is used.This method uses 4 or 8 connected neighboring pixels to evaluate the resulting image.The algorithm calculates a discrete image stemming from source image borders.
Identifying objects in the scene requires labeling each element as a plant and obtaining a regional description in order to extract features in the next step.Therefore, an algorithm based on connected components is used.The region labeling stage evaluates each pixel with a 4-neighbor connectivity, using a heuristic stated in pixel values according to predecessor labels at the north and west positions.Once objects on the scene are labeled, the next step is to extract area features from each element to discriminate weed and crop.The algorithm presented defines an area by counting the number of pixels in the object region, and then the value is stored for all items.The elements are sorted according to the area values in descending order.When a difference with the next object evaluation is greater than 50%, the average of the previous elements is calculated.This value is the threshold for weed detection.The classification is shown in Figure 3b.A complete description can be reviewed in (Pulido-Rojas, Molina-Villa, & Solaque-Guzmán, 2016).

Stereo vision system -3D point cloud
The main idea of the present work with stereo vision is to produce images from an RC coordinate system, predict a point cloud and determine the free work space.For this purpose, the ZED, a passive-depth camera, was utilized.Its availability to work in outdoor environments, light weight, reduced size and moderate cost allow a flexible mounting on aerial or terrestrial robots.In addition, this camera has a 110° field of view and can reach a high frame rate; reliance on NVIDIA graph cards (up to 100 Hz) and combination with an ROS framework make system implementation practical, with a high degree of integration for different robotic platforms and tasks.In this case, a zed-ros-wrapper package was used, which permits the recovery of depth and point clouds.Figure 3a and b.For the experiments, carried out on the campus of Universidad Militar Nueva Granada in Cajicá, Colombia), the camera was mounted in RC Robot position with sufficient space for an XYZ environment, in such a way that it acquired the necessary images to evaluate the cloud points in 3D.The sets of images that make up the motion sequence were acquired with the ZED camera connected with a USB 3.0 interface to a computer with a 2.6 GHz processor, 20 GB of RAM and NVIDIA GeForce GTX 960 M graphics card running Ubuntu 16.04 LTS.The ZED image processing results are presented in Figure 4.

A* algorithm, RRT algorithm and BEZIER curves
The approach to determining the weed tool trajectory was based on the A* and RRT algorithms.With the trajectory established, the Bezier theory is used to smooth the path.

A* Theory
The planning of the movement based on the algorithm A_star (A*), has as input parameters the initial position of the effector -Pi, the final position that corresponds to the location of the weeds -Pf, and the location of the obstacles.
A* has been used in the solution of planning problems for a long time.It is a relatively simple coding algorithm, quick to evaluate, and oriented towards global solutions.The principle of A* focuses on knowing the information of the scene (guaranteed by the previous procedures), bearing in mind the cost of moving to the next node and what remains to reach the final point, the sum of which is the real cost to advance.This can be described by equation 2, where: n is the node to be evaluated; g is the cumulative cost to reach that node from the starting node; and h is a heuristic value ascribed to moving from the node to be evaluated to the destination.A* is a complete algorithm; that is, it finds a solution as soon as it exists (LaValle, 2006).At the computational-cost level, it is proportional to the discretization of the workspace and the quality of the heuristics associated with the problem.

RRT Theory
The RRT algorithm has gained importance in trajectory planning due to its versatility and completeness.LaValle published the method in 1988, showing the basic principle of operation.In the following years, he presented improved variants of the initial algorithm (LaValle, 2006).
The basic principle of RRT focuses on the construction of a tree of possible configurations (qi) from the starting point (qstart).The tree must uniformly cover all configuration space that is free of obstacles.This is achieved by determining a random configuration (qrand) and extending the tree in that direction.This procedure is limited by an upper bound that marks the maximum iterations of the algorithm (Maxiterations) in case the final positions (qtarget) cannot be reached.Then, they are evaluated from the configuration nearby (qnear), which is the one that occupies the current position and from which the extension projection of the tree is split, with the possibility of moving to the new position (qnew).
If qrand is too far from a limit (ϵ), the configuration qnew is left in this direction, at distance ϵ.It is considered the new position; on the evaluation of a metric that is generally the Euclidean distance (ρ) and the one that lower cost represents, it will be the new position added to the tree.The procedure is followed until qtarget or Maxiterations are reached.

Bézier Theory
This theory is based on polynomial parametrizations and has been dealt with at length in the literature (McCartin, 1991).The principle is simple: if you want to join two points, it can be done with straight lines.If it is instead desired to join by means of a curve, anchoring points are placed that pull the line joining the two initial points toward these points, deforming the initial line.The extrapolation of these points produces polynomials that approach the anchor points, leaving the trajectory continuous.

Strategy for weed removal according to the Ceres AgroBot platform
Based on the systems that compose the platform, the trajectory planning that the final effector must follow was performed until weeds were reached.Starting from a depth map-the location of the objects of the scene (plants of interest, free space and weeds)-the objects of interest were encapsulated, giving a margin of safety subject to the size of the tool; Figure 5b illustrates the encapsulation, encompassing the objects in cubes that cannot be touched during RRT planning.From trajectory planning theory, these objects are considered obstacles in the configuration space.It should be noted that the idea of using a drilltype tool was based on studies for land conservation, as it is desirable to avoid chemicals or elements that damage the soil.Hence, mechanical weed destruction at stem and root was favored, whereby the destroyed weed then serves as fertilizer.

Results and discussion
After proposing the procedure and the way to solve the problem of finding the path to go from the starting point (qstart) to the end point (qtarget), two scenarios were posed (one with algorithm A* and another with RRT) to show the effectiveness of the solution proposed in this article and that fits the Ceres AgroBot robot.
Given that the artificial vision system delivers a discretized workspace in a point cloud where, after processing, it leaves the elements that must be protected marked (for the simulation points labeled as obstacles) and the points where the weeds are indicated (goal point), we proceed to apply the algorithms and subsequent smoothing by Bézier.

Scenario 1 -A*
Due to the lengths of the actuators and the size of the weed-removal tool, we have a workspace discretized at 5.This leaves 24 points on the  axis, 18 points on the  axis, and 12 points on the  axis.
With the result of the trajectory found by A*, this trajectory is passed by the Bézier algorithm, which in turn plays an implicit role in smoothing the curve and discretizing it.Thus, the trajectory is found in a period of 10 seconds, within which the actuators must carry the weed-removal tool (this time is calculated from the characterization of the linear actuators when they make their way).Figure 6 presents the resulting curves.Strong changes are observed over 2 to 4 seconds and from 7 to 8 seconds, a situation that is improved by the continuity achieved by the Bézier curves, allowing a more continuous movement by the actuators.The previous results and the control points for the determination of the Bézier curve are presented in Figure 6a.It should be noted that the tool is overlapping on the objects when it is smoothed with the result; however, it is covered by the tolerance zone that is left when closing the objects of interest (guard band).An analysis of times shows that the algorithms are fast in their execution, not exceeding the 3.1 seconds of the machine in an i7-processor system of 2.5 GHz.These algorithms run faster in the on-board system, since they are encoded in C and under Linux-ROS.

Scenario 2 -RRT
Under the same principle of the length of the linear actuators and the size of the tool, the workspace is discretized, and another scenario is proposed to apply the RRT algorithm.The results are shown in Figure 7.
The curves created by RRT present strong changes over 1 to 9 seconds, a situation improved by the Bézier curves.The execution time of the algorithms is 2.78 seconds.Figure 7a presents the results of the planning and smoothing of this curve in the same picture.

Conclusion
This paper shows a practical real system for weed detection and removal using artificial vision and path planning.The artificial vision used an area-based feature to discriminate from crops.This study was limited to periodical weed removal tasks, whereby weed size is smaller than the crop size to carry out the approach reported, achieving high sensitivity, specificity, and positive and negative predictive values.The path planification techniques were based on global searches such as A* and RRT, with path smoothing under Bézier theory.Similar results were found when both algorithms were tested, so that either works well for this application.

Figure 3 .
Figure 3. Images used in the identification of the position of the weeds.

Figure 4 .
Figure 4. 3D cloud point results: Images made over a plantation.

Figure 5 .
Figure 5. Scenario to perform the trajectory planning that the tool should follow.a) Scenario for image capture; b) Scenario for the encapsulation of the objects of interest.