Back in 2019, a Stanford professor and a student packed a car in California with stuffed penguins.
They drove to Santa Clara, to an area where hobbyists fly drones and radio controlled aircraft. They pulled out the penguins and scattered them around the ground. Then they flew drones, using special software to create paths for photographing the stuffed birds.
“I’m sure these hobbyists thought we were crazy,” laughs Mac Schwager, assistant professor of aeronautics and astronautics.
It may have looked unusual, but Schwager was involved with a serious task. Some of the lessons learned that sunny California day would ultimately be utilized with hundreds of thousands of real penguins, very close to the south pole. Specifically, drones would soon be flying above a barren patch of rock and ice known as the Cape Crozier Adélie Penguin Rookery, where hundreds of thousands of penguins gather annually to procreate.
There too, just like clockwork, a small research team comes to document and learn about that population. It’s fascinating work. But it’s hard to imagine a more remote part of the planet: Barren. Forlorn. Distant.
“There’s really no place like it on earth,” says Kunal Shah, a PhD candidate at the Stanford Multi-Robot Systems lab, or MSL.
“A lot of it is very untouched by humans, very raw.”
Just take a look:
There’s no electricity, no running water – very few of the comforts most of us rely on in our daily lives. But there is important science to be carried out: Studying those penguins. Specifically, trying to nail down precisely how many of these creatures nest here during the antarctic summer – roughly October through February – and whether there might be a correlation between population fluctuations and the shrinking seasonal levels of frozen sea-ice.
That’s a huge job. And drones are now playing a very big role in the research. But it wasn’t always this way.
The old method
Traditionally, scientists had but one option: Manually tagging individual penguins – say about 1,000 out of the entire colony – and then finding as many of those penguins as possible every subsequent year. Based on the penguins spotted and the density of nests and the area of the overall rookery, you’d be able to draw some data-driven conclusions about the total population numbers.
Was it perfect? No. But it was the best available option. Helicopters are too expensive, noisy and intrusive for aerial surveys. Plus, they would have to be flown below legal altitudes to capture the necessary resolution for the images to be truly valuable.
So that was the standard: Tedious, time-consuming tagging and counting.
At least that was the case until a serendipitous conversation between two strangers at a California wedding.
What are the odds?
Back in June, 2016, there was a chance meeting at a wedding. Biologist Annie Schmidt, whose research at Point Blue Conservation Science focuses on penguin ecology, happened to meet Mac Schwager. Mac is an assistant professor of aeronautics and astronautics and Stanford. Some of his work involves capturing data via drone.
Soon, Annie was describing the challenges of the existing methodology for that South Pole field work. Mac talked about the potential benefits of using drones: They could provide non-invasive, high quality data at a relatively inexpensive cost. With the right approach, drones could not only save scientists the drudgery of finding and counting tagged penguins – they could provide an excellent data set in a fraction of the time with a sliver of the hassle.
The pair wrote a grant proposal; the National Science Foundation agreed to fund it.
Enter DJI
The grant was written back in 2017, before DJI announced its Inspire 2 and M200 drones. At the time, the most suitable product was the Matrice 100. The team picked up five of these machines.
M100 pre-flight: One of the drones used by the researchers to survey the penguin colonies. The drones operated autonomously and collaboratively by running off an advanced path planning algorithm.
(Image credit: Parker Levinson/ Point Blue)
But that’s only the beginning of this remarkable story. Because the team would soon be working with PhD student Kunal Shah. Before long, Kunal would be tweaking code, trying to optimize an algorithm for the task. The goal? To ensure that multiple drones, flying simultaneously but on different paths, could capture the entire penguin population at the rookery in the most efficient manner possible.
“It’s basically to make the best use of limited flight time,” says Shah.
Why the concern about speed and efficiency?
Windows providing suitable weather for drone flights are often brief. Cold temperatures can impact battery life. The batteries are charged via solar power, which takes time. Researchers wanted to ensure minimal disruption for the penguins. And – big factor here – penguins are often on the move. The faster the photos can be collected, the more likely researchers won’t be counting the same penguin(s) more than once.
Well, that’s simple. Just fly a grid, right?
Nope. Not at all.
An amazing algorithm…
Let’s pretend you’ve dropped something valuable on a lawn. Further, imagine there’s but one access point in or out of the yard. Entering, your instinct would be to search in a grid pattern, walking careful rows to ensure you get a chance to see everything on the ground.
Most of us – and even some drone software – would apply the same logic to photogrammetry flights. We’d be inclined to fly a neat grid, stopping and taking photos at each waypoint. And, obviously, each of those waypoints would be on a row. Take one row of images, then another, then another...until you’re done. Think of it as mowing a lawn in the sky.
Believe it or not, this is *not* the most efficient way to cover that territory from the air. And, once you factor multiple drones into the equation, it’s definitely not the best method.
That’s where Kunal Shah comes in. He’s now been working with robotic path planning for five years – looking at ways to calculate the most efficient paths machines with certain tasks can take. He’s also a drone pilot.
Mac Schwager asked if he was interested in participating in the project. Shah recalls the conversation:
"Hey – do you want to go to Antarctica? You’ve got a week to think about it."
As you likely guessed, Shah said “Yes.”
How does it work?
Picture a map of that rookery – a rugged area with many hundreds of thousands of penguins. Now imagine that you’ve determined the optimum altitude for taking photographs and how close each photo should be for sufficient stitching overlap. You’d wind up with a series of waypoints that looks like an overlaid grid.
Shah’s algorithm, however, looks at this problem in a different way. It calculates the most efficient flight paths for each of multiple flying robots. It ensures that a photo is taken at each waypoint, but that there’s minimal backtracking over the whole flight path.
“The idea behind this is that we wanted to minimize the amount of backtracking the drone is doing. If you look at a lot of the classic ways of solving this problem, you have some area and you’re basically flying back and forth over this path. But often what happens is the starting point and ending point are geographically very far,” explains Kunal.
“And this is not really much of an issue if the path is a couple of acres of farmland or the living room for your robotic vacuum - the backtracking path might take only a couple of seconds to traverse. But in a large area like this, the return path from the beginning to the end could be on the order of a kilometer. And that’s wasted battery life, right? If you planned better, you could use that battery life and ideally stop where you started.”
This isn’t something you could calculate on your own. Far from it.
“As you add more points, the computational complexity skyrockets,” says Stanford’s Mac Schwager.
And the paths produced? Have a look:
The roads less travelled. These are the kinds of paths created by the algorithm to ensure speed and efficiency...
The algorithm takes into account the number of photos required, the area, the flight time, and the number of drones involved (they had up to four M100s in the air at once). It then plots a path through those waypoints with a common beginning and endpoint. The result? An intricate series of zigs and zags that ensures all photos will be taken – without any repetition – all while maximizing battery life.
"It tends to produce patterns that look like fractals – kinky and convoluted,” says Mac Schwager. "There’s no common back-and-forth lawn mower pattern. That kinkiness is valuable.”
Even if you were flying a single drone, the algorithm would produce a path very different from – and more efficient than – a regular row-on-row flight.
Better than a single drone
The year before Kunal was on the project (he was at the South Pole from the end of November 2019 until early February of 2020), the team tried flying a Phantom 4 Professional in a grid pattern. Success using that lawn mower path was limited. Batteries had to be swapped out during the mission. And, because the overall flight took longer, there was the problem of penguins moving potentially significant distances, showing up in more than one photo, and being counted twice (or more).
With one pilot and one drone, this work took more than two days. With multiple M100s and the algorithm, it took about three hours. So there was a huge difference in efficiency.
“This was sort of the first time this was ever done,” says Shah.
And this...is what it looked like:
Lots and lots of data
With the M100s flying at 50 meters, the resulting image resolution was roughly 1.5 cm/pixel. With a 60 percent overlap, the area required between 2,000 and 3,000 photos – resulting in single aggregate images of up to 10 gigabytes.
“Those photos were all uploaded to stitching software that creates a giant photomosaic,” says Shah. “And then that image is sent into a machine learning pipeline that will classify what is a penguin – and will automatically count the penguins in that massive image.”
The surveys would produce thousands of images, stitched together to produce dense, high-resolution images like this one.
Research report
This was cutting-edge work, and it resulted in a paper published in Science Robotics. The abstract of that paper summed up the huge difference in efficiency when using multiple drones and the algorithm:
“Over 2 square kilometers was surveyed in about 3 hours. In contrast, previous human-piloted single-drone surveys of the same colony required over 2 days to complete. Our method reduces survey time by limiting redundant travel while also allowing for safe recall of the drones at any time during the survey. Our approach can be applied to other domains, such as wildfire surveys in high-risk weather conditions or disaster response.”
And that last bit is an important point. Mac Schwager says this algorithm has clear use-case scenarios with other types of data collection using multiple sensors. Really, he says, it could be employed anywhere there’s a need to gather accurate data quickly and efficiently.
“The overall advantage,” says Professor Schwager, “is speed.”
With projects like this, we’re seeing increasing applications where drones, coupled with AI and automation, are doing very new things. Schwager says this is really just the beginning.
“This is a revolution in all areas of engineering research,” he says. “There’s no doubt about it.”
Ross locations: Geographic locations of Cape Royds and Cape Crozier on Ross Island, Antarctica. (Image credit: Annie Schmidt/ Point Blue)
How did the M100s perform?
Despite not having the heated batteries that came with subsequent models, the M100s were up to the task. The team kept the batteries warm until just prior to the missions.
“They’re well-built machines,” says Shah. “And I was pleasantly surprised.”
Another research team went back for the 2020-2021 Antarctic summer. Shah did not join them, at least partly due to COVID-19.
“I was absolutely supposed to go back this year, but the world sort of exploded,” he explains. “I might go back next year (2021-2022); I would like to keep working with this group of people.”
Plus, there’s always the penguins.
“They’re not habituated to humans, so they’re pretty curious. It was a very nice privilege to go observe them,” says Shah.
A curious Adélie penguin (Image credit: Annie Schmidt/ Point Blue)
“They’re still wild animals, but they’re curious and they’ll come up and walk with you.”
That’s at least partly because – unlike the drones – penguins can’t fly.
Additional Materials:
- https://www.sciencemag.org/news/2020/11/watch-speedy-drones-count-antarctic-penguin-colonies-record-time%20
- https://news.stanford.edu/2020/10/28/multi-drone-system-autonomously-surveys-penguin-colonies
- https://www.youtube.com/channel/UC-EnprmCZ3OXyAoG7vjVNCA
- https://www.youtube.com/watch?v=h1YXXWj2E4w
