Software apps and online services
Containment Sentinel is a force multiplier extending perception of those responsible for maintenance of firebreaks. Simple to use with an off-the-shelf interface, anyone with google map skills has the expertise required to operate the Containment Sentinel. Autonomous flight modes meet core project requirements enabling operators to patrol a firebreak on autopilot without ever using the included manual flight controller. This example video the Containment Sentinel executed all flight maneuvers autonomously.
Rapid Deployment.The Containment Sentinel is simple to use, when it boots up it is ready to monitor your fire breaks. Simply define an area of responsibility using the mission planner in QGroundControl. After uploading the mission, allow the Sentinel to take off and patrol.As the Sentinel flies along the the waypoint defined fire break it is constantly scanning the ground with a gyro stabilized sensor package. The stabilizer is an off-the-shelf 3-Axis gimbal known as a STorM32BGC.
Here is a screen shot of the QGroundControl application displaying a mission waypoint list.
The following file is a simulated file where I added “hotspots for the application to display. I Place the waypoints and “hotspots” on city streets so judges can quickly correlate the hotspots map with the waypoint map. This is not live data as I did not fly the drone through the town.This mission flight path circumscribes a wheat field near my place of employment. The field occasionally catches fire when the discing machines strike rocks prompting the fire department to evacuate our office. During an incident, the Containment Sentinels fly the waypoints looking for sparks jumping the natural firebreaks. The winds here are predominantly westerly so I added hotspots to the log file indicating there is a breach onto the neighboring field to the east. Note: the waypoint line below was added after the fact for clarity. real time processing is possible with mission download.Presented with this data before smoke is even visible, the fire fighters quench this fire before it flares up.
Enabling the application on your computer.
The Sensors.This version of the Sentinel’s sensor package has a FLIR Lepton 3.5 thermal camera and a PiCamera for optical scanning. The FLIR camera has a Field of View (or FOV)of 56deg. This works out to offer a horizontal view of 10.6 feet if the drone if flying 10 feet off the ground. The PiCamera module has a 45deg FOV. Toward the end of this article I discuss live testing to illustrate that the FLIR camera detects heat extremely well. The Optical PiCamera is there to offer context for the thermal images. Understanding what is around a hotspot is key risk analysis.
Hotspot detection.Hotspot detection is achieved in real time through onboard image processing using OpenCV on a Raspberry Pi 3. The FLIR camera and breakout board provide a UVCvideo (USB Video Class) interface that OpenCV can access as a slow video feed. As the Sentinel patrols the fire line the FLIR camera sees mostly dark/cool pixels. However, as a hotspot or a spark crosses its field of view OpenCV classifies the hotspot.
Hotspot Classification.OpenCV processes the images in real time measuring the relative intensity and location of the hotspots. OpenCV processes the thermal image by running a binary threshold to separate the background from the hotspots.
Typically the Sentinel expects the left side of the image to be the cold side, so any hotspots there are logged. Ongoing development will provide the ability to dynamically define the “hotside” of the firebreak by comparing the waypoint line and azimuth (or direction) to a rallypoint. Logging hotspots.Given that all hotspot breaches are of interest, the Sentinel saves the raw thermal image and an annotated thermal image for later review. The annotations are hotspot markers, Region of Interest (ROI) and camera heading. An additional optical image may be saved under only certain circumstances to prevent oversaturating the operators with similar images. Initially the Sentinel is configured to capture an optical image if there is a hotspot and it hasn’t saved an image in the last 10 seconds. Other criteria could be location-based such as field of view lockouts. These would inhibit the camera from capturing a new image if the FOV overlaps an image already stored.
The Raspberry Pi is running a desktop where the inspection system displays live data with live images. A remote desktop connection provides access to the real time operations on the Raspberry Pi. Below is a screenshot of the desktop. These images from the sparkler test described later.
Below are the three images: raw thermal, annotated thermal, and optical.
Here is a video of the flight taken while collecting these images.It was in manual flight mode due to space constraints and to collect as much data as possible.
Log file.All hotspots on the cold side of the fire break are logged in a Comma Separated Variable file (or.csv). This file saves the latitude, longitude, size, date, time, image notifications, and relative altitude for later analysis.
Live Testing.Contained candles on rain soaked green grass were employed as stand-ins for actual hotspots. The Sentinel hovered over the flame sources on both the hot and cold side of the detection range. A 4th of July sparkler was used to illustrate the wide range of temperatures the hotspot classifier is capable of detecting. Next to the sparkler is one of the candles which was blown out earlier.
Here is a video of the flight taken while collecting these images. It was in manual flight mode due to space constraints and to collect as much data as possible.
To the naked eye the candle appeared cool and solid. However the thermal camera detected the residual heat. This is similar to a smoldering pine cone waiting to re-lite a previously controlled area through the simple addition of wind. One firefighter mentioned a containment breach caused by moss. To the naked eye the moss was cool, but the thermal image would have exposed the underlying embers. Also, while testing high sensitivity parameters, footprint thermal blooms were clearly visible contrasted against the cool wood floor.
Further testing, in a safe area, to see how sensitive the thermal images were yielded promising results. I used three hand warmer packs sitting on the snow: two in one pile and one off by itself. In the image below from an altitude of 15 meters, the two heat packs at the green arrow and the single at the orange arrow are still visible.
The two heat packs indicated by the green arrow are still visible at an altitude of 43 meters.
Here is an image from the optical camera, at 27meters, for comparison.
I plan to continue working on the thermal camera to increase thermal discernment through use of a histogram rather than the auto gain control method.
Containment Sentinel's wifi card operates in a mesh networking topology. This self healing coverage allows an IOT medallion notification interfaces to run in low power mode extending battery life. As the network shares data about locations and hotspot status, a Containment Sentinel notifies the closest operator if a hotspot meets certain alarm criteria. Once the operator deals with the hotspot, a IOT button input updates a hotspot status on the network. The self healing nature of meshnets ensure multiple paths for data to reach the command center.
Further Meshnet and IOT benefits.A firefighter will use the IOT medallion to request the nearest drone fly-by their location tasking its cameras on their work area. The medallion's built in GPS enables orbit loitering around a firefighter upon request.
If a hotspot is too large or spreading too quickly for shovels, the command center has the option to deploy a fixed wing drone carrying fire retardant. Conventional retardant bombers require trained pilots and carpet large areas. Autonomous fixed wing aircraft remain ready to deploy at a minutes notice, and provide the commander options to deliver targeted amounts of fire retardant.
IOT burnover supportIn the case a firefighting crew must weather a burnover, Containment Sentinels provide situational awareness to the crews inside their shelters. Burnover events are when a crew is trapped in the path of a fire, they deploy protective tents and wait for the fire to pass over. Quick reaction retardant drones can also zero in on the firefighters' IOT medallions even in high risk flight conditions.