SYSTEM AND METHOD FOR COLLABORATIVE UAV AND UGV-BASED MEDICAL SUPPLY TRANSPORT USING REINFORCEMENT LEARNING

Abstract

A reinforcement learning-based framework for collaborative UAV-UGV transport in healthcare is disclosed. The invention includes autonomous UAV and UGV units equipped with navigation systems, encrypted communication, and a centralized control module, optimizing medical supply delivery. The framework employs a multi-agent reinforcement learning model for path optimization, collision avoidance, and real-time adjustments, ensuring efficient and timely medical logistics.

Specification

Description:FIELD OF INVENTION
The present invention relates to a coordinated framework involving Unmanned Aerial Vehicles (UAVs) and Unmanned Ground Vehicles (UGVs) for the autonomous and efficient transport of medical supplies in healthcare settings.
More specifically, it utilizes reinforcement learning algorithms and secure communication systems to optimize route planning and collision-free coordination between UAVs and UGVs.
More particularly, the present invention is related to System and Method for Collaborative UAV and UGV-Based Medical Supply Transport Using Reinforcement Learning.
BACKGROUND & PRIOR ART
The subject matter discussed in the background section should not be assumed to be prior art merely as a result of its mention in the background section. Similarly, a problem mentioned in the background section or associated with the subject matter of the background section should not be assumed to have been previously recognized in the prior art. The subject matter in the background section merely represents different approaches, which in-and-of-themselves may also be inventions.
In recent years, the demand for rapid and efficient delivery of medical supplies, especially in critical and remote healthcare situations, has increased significantly. Conventional transport methods often face limitations due to geographical constraints, traffic congestion, or infrastructure challenges. Autonomous vehicles, such as UAVs and UGVs, provide a promising solution by leveraging aerial and ground capabilities to improve the efficiency and speed of supply chains in healthcare.
However, coordinating UAVs and UGVs in dynamic environments, such as urban or rural areas, presents challenges, including collision risks, communication lags, and optimization of travel routes. Traditional approaches to UAV-UGV coordination often lack the flexibility and adaptability required in uncertain environments. Therefore, there is a need for an optimized, autonomous framework that allows UAVs and UGVs to collaborate effectively, thereby enhancing delivery reliability and speed in healthcare applications.
As used in the description herein and throughout the claims that follow, the meaning of "a," "an," and "the" includes plural reference unless the context clearly dictates otherwise. Also, as used in the description herein, the meaning of "in" includes "in" and "on" unless the context clearly dictates otherwise.
The use of any and all examples, or exemplary language (e.g. "Such as") provided with respect to certain embodiments herein is intended merely to better illuminate the invention and does not pose a limitation on the scope of the invention otherwise claimed. No language in the specification should be construed as indicating any non-claimed element essential to the practice of the invention.
The above information disclosed in this Background section is only for the enhancement of understanding of the background of the invention and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art.
OBJECTIVE OF THE INVENTION
The principal objective of the present invention is to present a system and method for collaborative UAV and UGV-based medical supply transport using reinforcement learning . .
The objective of this invention is to establish an autonomous, collaborative UAV-UGV system that efficiently transports medical supplies by optimizing route planning, enhancing collision avoidance, and ensuring secure, real-time communication. This system aims to improve the speed, reliability, and cost-effectiveness of healthcare logistics, especially in remote and critical settings.
SUMMARY
Before the present systems and methods, are described, it is to be understood that this application is not limited to the particular systems, and methodologies described, as there can be multiple possible embodiments which are not expressly illustrated in the present disclosure. It is also to be understood that the terminology used in the description is for the purpose of describing the particular versions or embodiments only and is not intended to limit the scope of the present application.
The present invention provides a reinforcement learning-based UAV-UGV collaborative framework designed for the efficient and autonomous transport of medical supplies. The system includes:
UAVs equipped with autonomous navigation, reinforcement learning modules for optimal path selection, and real-time communication systems.
UGVs with autonomous ground navigation, collision avoidance capabilities, and reinforcement learning-based route planning.
Centralized Control System for monitoring, supervising, and dynamically adjusting the paths of both UAVs and UGVs.
Encrypted Communication System facilitating secure data exchange between UAV and UGV.
The invention enables UAVs and UGVs to work in tandem, optimizing both aerial and ground routes through reinforcement learning-based algorithms. The system dynamically adjusts paths based on real-time environmental data and a multi-agent reinforcement learning model, minimizing travel costs and improving delivery efficiency.
BRIEF DESCRIPTION OF DRAWINGS

To clarify various aspects of some example embodiments of the present invention, a more particular description of the invention will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. It is appreciated that these drawings depict only illustrated embodiments of the invention and are therefore not to be considered limiting in its scope. The invention will be described and explained with additional specificity and detail through the use of the accompanying drawings. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure:
Figure 1 shows block diagram representation of System for Collaborative UAV and UGV-Based Medical Supply Transport Using Reinforcement Learning.
DETAIL DESCRIPTION

The present invention is related to System for Collaborative UAV and UGV-Based Medical Supply Transport Using Reinforcement Learning.
Figure 1 shows block diagram representation of System for Collaborative UAV and UGV-Based Medical Supply Transport Using Reinforcement Learning.
This invention introduces a novel reinforcement learning-based framework designed to coordinate Unmanned Aerial Vehicles (UAVs) and Unmanned Ground Vehicles (UGVs) for the efficient and secure transport of medical supplies within healthcare environments. The framework leverages the strengths of both UAVs and UGVs, where the UAV provides rapid aerial transport and the UGV handles ground navigation, enabling a seamless flow of goods across diverse terrains and challenging environments. The primary focus of this invention is to overcome logistical obstacles in healthcare supply chains, especially in remote or underserved areas, by utilizing a cooperative, autonomous approach that improves delivery speed, reliability, and cost-effectiveness.
In this framework, UAVs are equipped with advanced navigation systems that allow for autonomous aerial movement through GPS, LIDAR, and computer vision-based guidance. This system supports real-time altitude and route adjustments, allowing the UAV to avoid obstacles, optimize flight paths, and conserve energy by maintaining low altitudes whenever feasible. The UAV is also equipped with a reinforcement learning module that enables it to calculate optimal waypoints from start to target locations, factoring in 3D coordinates and environmental conditions. By employing reinforcement learning algorithms, the UAV can dynamically adjust its route, minimizing travel costs and reducing potential collision risks in mid-air navigation.
The UGV, in turn, provides a ground-based counterpart that complements the UAV's aerial capabilities. Equipped with GPS and LIDAR systems, the UGV autonomously navigates roads and pathways, optimizing its route through complex terrain. Similar to the UAV, the UGV is also integrated with reinforcement learning mechanisms that enable it to adapt its path based on real-time environmental data, such as obstacles and traffic conditions. The UGV's route is carefully calculated to ensure minimal travel cost, and its movement is continuously monitored to avoid collision zones, making it ideal for traversing both urban and rural healthcare settings. Together, the UAV and UGV form a collaborative transport system, each vehicle utilizing its specialized capabilities to overcome the other's limitations, thereby ensuring smooth and coordinated delivery of medical supplies.
To facilitate seamless coordination between the UAV and UGV, a secure communication system is implemented. This system enables real-time data exchange through encrypted wireless networks, satellite links, or other dedicated communication protocols, ensuring data integrity and security across all transport phases. The secure communication link allows the UAV and UGV to share position, route, and environmental data, enabling them to make informed decisions based on each other's status and position. This real-time communication is critical for maintaining synchronized movements and ensuring that each vehicle's route is optimized based on dynamic environmental factors.
At the core of this framework is a centralized control system that oversees and supervises the operations of both the UAV and UGV. The centralized control module monitors each vehicle's position, adjusting routes dynamically when necessary to optimize the efficiency and safety of the transport process. This control system provides the overall operational intelligence, directing the UAV and UGV to work in tandem, adapt to changing conditions, and respond to unforeseen obstacles. For instance, if the UAV encounters an unexpected obstacle during flight, the control system can re-route the UGV to intercept at an alternative location, thus ensuring delivery continuity. This centralized control system is essential for maintaining coordinated, efficient, and safe medical transport.
One of the key innovations in this framework is the application of reinforcement learning to enable the UAV and UGV to maximize their cumulative rewards over a series of actions and environmental states. By applying the Bellman equation, each vehicle learns to select actions that lead to optimal outcomes over time. Each vehicle receives immediate rewards for successful actions that avoid obstacles and follow efficient paths, while penalties are applied for actions leading to collisions or inefficient detours. This reinforcement learning approach enables the UAV and UGV to autonomously select the best routes under varying conditions, minimizing travel time and energy expenditure.
In addition to route optimization, this framework is designed to enhance cost-effectiveness and energy efficiency. The UAV's altitude is optimized to reduce energy costs, keeping it at lower altitudes when possible to conserve power while avoiding ground obstacles. The UGV's path is planned with a focus on minimizing the number of waypoints, which helps to reduce operational costs associated with long or complex routes. By leveraging reinforcement learning to optimize both altitude and ground path costs, the framework ensures an energy-efficient delivery process that can be scaled to various healthcare scenarios without compromising reliability or speed.
This invention provides a transformative approach to healthcare logistics by combining the aerial reach of UAVs with the ground resilience of UGVs in a synchronized, learning-driven system. By addressing critical challenges such as autonomous path planning, collision avoidance, secure communication, and centralized control, this framework significantly improves the reliability, speed, and efficiency of medical supply transport, making it an ideal solution for healthcare delivery in both routine and emergency settings.
The present system includes:
A UAV equipped with autonomous navigation, LIDAR, GPS, and computer vision-based guidance systems, allowing it to navigate complex aerial routes efficiently. The UAV operates at varied altitudes to minimize fuel or battery consumption, adapting to environmental conditions to ensure the safety and timeliness of medical deliveries.
A UGV capable of traversing ground routes with autonomous navigation systems such as GPS and LIDAR, allowing it to navigate through roads and terrains that complement the UAV's aerial path.
Centralized Control System responsible for overseeing UAV-UGV coordination, facilitating real-time adjustments based on environmental changes, and ensuring that the UAV and UGV collaborate effectively.
Communication System that securely exchanges data between UAV and UGV. The communication link is encrypted to ensure data security, leveraging wireless networks, satellite links, or dedicated protocols.
The UAV and UGV employ autonomous navigation systems guided by reinforcement learning algorithms for collision-free and optimized path planning. The reinforcement learning model employs a reward system that encourages the UAV and UGV to choose paths with minimal travel costs while penalizing collision-prone routes.
Path planning is executed as follows:
The UAV selects a series of waypoints defined by 3D coordinates to navigate from the start to the target point. The altitude cost and distance between waypoints are calculated to ensure efficient path planning.
The UGV plans its route by minimizing ground path costs based on the number of waypoints and the absolute distance between waypoints.
Reinforcement Learning-Based Coordination:
The UAV and UGV collaboration is facilitated by a reinforcement learning framework with the following components:
State (S): Represents the environmental conditions, waypoints, and the UAV-UGV positions.
Action (A): The steering actions chosen by the UAV and UGV for optimized movement.
Reward (R): Calculated based on the path length, collision avoidance, and efficient resource utilization.
The system employs the Bellman equation to update the learning process and selects actions that maximize cumulative rewards over time, enhancing UAV-UGV route coordination.
The centralized control system continuously monitors UAV and UGV locations, adjusting routes dynamically. The communication system enables secure and real-time data exchange, supporting a distributed decision-making process where the UAV and UGV adjust routes based on each other's positions and environmental factors.
The UAV-UGV framework incorporates a dynamic routing protocol that predicts and adjusts paths to avoid collisions. For the UAV, a cost function based on altitude is considered to ensure energy-efficient flights at lower altitudes. The UGV's route planning incorporates steering mechanisms that minimize ground path cost and avoid collision zones.
.Coordinated movement of UAV and UGV
Drone path planning considers the zones at different altitudes with a total n number of waypoints H=(h_1,h_2,……,h_n ). Each waypoint is defined by the three 3-D coordinates h_i,=(x_i, y_i,z_i),i=1,…..,n . An objective is to decide a path comprising optimal waypoints (h_s,h_1,……,h_n ) comprising starting point, different waypoints (h_i,h_(i+1))?H
and termination point. The following criteria are considered in path planning,
UGV Steering
The cost of region is defined with the waypoints of the path as the more number of waypoints, the higher the cost of zones and the cost of region is defined as shown below.
R_ugv (t)=1/(|h_s,h_T |) ?_(t=1)^T¦?|h_i,? h_(i+1) | (1)
where |h_i,h_(i+1) | be the absolute distance between two waypoints (i,i+1).
UAV steering
Cost of Altitude of the path, the cost will be higher if the altitude of the path is higher. At lower altitudes, the drone may fly with improved efficiency. The cost of altitude in terms of route length is defined as follows.
R_uav (t)=1/|z_max-z_min | ?_(t=1)^T¦?|(z_m ? (h_i,h_(i+1) )-z_min| (2)
where z_m (h_i,h_(i+1) ) is the mean height of flying at segments (h_i,h_(i+1) ) . z_max and z_min are maximum and minimum altitude, respectively. The following algorithm has been proposed for the coordinated steering of UAV and UGV.
Start
Input : different waypoints
Output : optimal path of UAV-UGV coordination
Predict set 3-D coordinates (h_s,h_1,……,h_n ) for optimal path in uncertain environment
h_(i+1)?Steer UGV (h_i,h_(i+1) ) and steer UAV (h_i,h_(i+1) )
Check mapping (?h_i,h?_(i+1))
Return True ( h_(i+1))
End ______________________________________________________________________________
REINFORCEMENT LEARNING For UAV-UGV FRAMEWORK
Let S and A represent a discrete set of environment state and set of actions, respectively. In every state s?S a framework of multi-drone mobile robot takes feasible action, ?A , over the finite learning horizon . The framework transits to the next state s^1?S and the framework receives immediate reward for performing desired action . However for each desired action leading to the collision, the framework is penalized. The goal of the framework is to maximize the reward. It does this by learning which action is optimal for each state. It does this by learning which action is optimal for each state. With available information of <s,a,s^',r_t>, the Q learning process is updated with following equation .


Q(s_t,a_t )?Q(s_t,a_t )+a_t (s_t,a_t )×
?[R?_(t+1)+? max-(a_(t+1) )??Q(s_(t+1) ? ,a_(t+1))-Q(s_t,a_t)] (3)

where a_t (s_t,a_t ) (0<a=1) is learning rate, R_(t+1) is the reward observed after performing a_t in s_t , the learning rate may be the same for all pairs, the discount factor ? is such that 0=?<1 . The Bellman equation that describes the optimal action-value function. Q^* (s,a) that described as given below.


Q^* (s,a)=E[r(s,a)+? max-(a^' )??(s^' ?,a^')] (4)

Where s^'~P represents the next state and that is sampled by the environment from a distribution P(.|s,a). This bellman function starts the learning approach of approximately Q^* (s,a). The set (s,a,r^',s,d)?D represents the transition and d tells about terminal of s^'. To remain Q^* (s,a) close to Bellman equation , a mean square Bellman equation is defined as given below.
L(Q^*,d)=E-((s,a,r^',s,d)?D)?[ Q^* (s,a)-(r+?(1-d)max-(a^' )??Q^* (s^',a^' )^2]? (5)

Where d equals 1 if s^' terminal else it is zero. In the target network the target term is defined is defined as given below.
r+?(1-d) max-(a^' )??Q^* (s,a)? (6)

By minimizing the MSBE loss it is possible to track this target by Q^* (s,a) , An optimal action a^* can be obtained by solving the following equation

a^*=arg min-a??Q^* ? (s,a) (7)

where a=[v_mr,v_dr ]^T The reward function for the optimal trajectory of UAV is as follows.

?rw?_uav (t)=?_d ?R_uav^p (t)-R_uav (t)?/?R_uav^p (t)? (8)

where R_uav^p (t) and R_uav (t) be the predicted optimal and actual route length of the drone and ?_d be the scaling factor. The reward function for the UGV is defined as shown below.

?rw?_ugv (t)=?_d ?R_ugv^p (t)-R_ugv (t)?/?R_ugv^p (t)? (9)

The total reward function is defined as shown below

rw(t)=?rw?_uav (t)+?rw?_ugv (t) (10)

Although implementations of the invention have been described in a language specific to structural features and/or methods, it is to be understood that the appended claims are not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as examples of implementations of the invention.
, Claims:
1. A coordinated UAV-UGV framework system for medical supply transport in healthcare, comprising:
o a UAV equipped with autonomous navigation, real-time data-sharing capabilities, and a reinforcement learning module for determining optimal flight paths, enabling efficient aerial transport;
o a UGV equipped with autonomous navigation, path prediction, and reinforcement learning mechanisms for collision-free movement on the ground to predetermined waypoints;
o a communication system for secure and real-time data exchange between the UAV and UGV; and
o a centralized control system to oversee UAV-UGV coordination, monitoring, and dynamic route adjustments, wherein the centralized control system comprising an optimal steering control that incorporates horizontal and vertical movement for collision avoidance, enhancing the stability and efficiency of the UAV-UGV collaboration, and a reinforcement learning mechanism, which comprises a Bellman-based update function to refine path optimization, accounting for variable terrain, environmental factors, and real-time position changes of both UAV and UGV.
2. The system of claim 1, wherein the UAV and UGV communicate via encrypted protocols selected from wireless networks, satellite links, or dedicated protocols for enhanced security and data integrity.
3. The system of claim 1, wherein the UAV-UGV coordination system dynamically predicts and adjusts routes based on environmental states and conditions through a multi-agent reinforcement learning approach, minimizing travel cost and ensuring delivery efficiency.
4. A method for coordinated UAV-UGV-based medical supply transport in healthcare, comprising the steps of:
o initializing UAV and UGV start and target locations, identifying waypoints in a 3D coordinate system;
o calculating an optimal UAV-UGV path by applying a reinforcement learning-based algorithm to steer each vehicle autonomously and avoid collisions;
o predicting and adjusting routes through a transfer learning mechanism that incorporates multinomial regression for estimating the optimal route length for the UGV and UAV;
o executing real-time data sharing between UAV and UGV through an encrypted communication system; and
o monitoring the progress and status of UAV and UGV via a centralized command system, adjusting paths as required to accommodate real-time conditions.
5. The method of claim 4, wherein path optimization includes steering adjustments based on altitude zones for the UAV, minimizing altitude costs by maintaining lower altitudes where feasible to improve energy efficiency.
6. The method of claim 4, wherein the reinforcement learning algorithm maximizes reward and penalizes collision-prone actions by updating the Bellman function to refine decision-making over the learning horizon.
7. The method of claim 4, further comprising the step of defining a reward function for the UAV and UGV based on predicted and actual route lengths, scaling the reward to optimize alignment between predicted and actual routes.
8. The method of claim 4, wherein the centralized control system continuously monitors and provides feedback to the UAV and UGV, enabling dynamic rerouting to ensure efficient and timely delivery.

Documents