The Long Wait

There was a long delay between Flight 52 and Flight 53, and a deviation from the published Flight 53 plan. I will address both in this blog post. First, the Flight 53 delay was the direct result of a decision by the team to fly Ingenuity out of telecom communications range with the rover. One may question the wisdom of this, but there are good reasons to do so.

When plotting a traversal path through a given area, the team identifies a series of smooth, flat regions referred to as “lily pads” where the helicopter can safely land without significant risk of damage to the vehicle. Because the helicopter’s laser altimeter is fairly sensitive to large terrain variations, the team also generally tries to plan flight paths over the smoothest and flattest terrain available. These terrain types also happen to be the areas favored by rover planners as they are more easily traversable and contain fewer obstacles that might impede rover progress. The result is that in difficult terrain, the two vehicles generally compete for the same narrow path.

Roughly speaking, the Ingenuity team’s top mission priorities are:

1 – Avoid significant interference with, or delay of, rover operations

2 – Maintain vehicle health and safety

3 – Perform scouting for tactical planning and science assessment

4 – Perform experiments to inform mission and vehicle design for future Mars rotorcraft, or collect data for discretionary science

In difficult terrain, the first two priorities result in an emphasis on staying well ahead of Perseverance. Staying only slightly ahead of the rover while providing good data transfer rates is risky since the helicopter may inadvertently block the rover’s path. Staying just behind the rover is also a challenge, as the flight paths of the helicopter would need to be too close to the rover for safety or incorporate energy-sapping diverts to maintain adequate distance. Staying to the side of the rover is often not possible due to communications and landing-site constraints. Staying far behind the rover obviously risks loss of telecom, which would force the rover to reverse (violating priority #1) or leave the helicopter behind (violating priority #2). Thus, the helicopter frequently operates in a narrow slice of terrain several hundred meters ahead of the rover’s position. When telecom permits, the team will try to land at locations parallel to Perseverance’s planned strategic route, but this has been the exception, not the rule over the last year of the mission.

This balancing act require careful planning to maximize operational flexibility, agile replanning of flights/activities on short notice, and a high number of operations shifts focusing on file transfers to compensate for the poor data transfer speeds at the fringes of the radio’s range. Slow data transfer turnaround reduces the achievable flight cadence and operational effectiveness of the helicopter.

Flying slightly ahead of telecom range is one strategy to cope with this environment, as it buys time, reducing the risk of rover interference and/or reducing the demand on the operations team to execute short-notice flights. This, however, has some drawbacks: 1) It further retards operation cadence, as it forces the helicopter to spend more time in regions with poor telecom. 2) It can potentially strand the helicopter beyond comms range if the rover encounters any delay or makes tactical decisions that deviate from the strategic route. This latter vulnerability came into play almost immediately after Ingenuity executed its 52nd flight.

A Waiting Game

Flight 52 was planned as a long-distance, out-of-telecom flight, which was executed on Sol 776 with the expectation that the rover was nearing the end of its exploration at Echo Creek and would be approaching the helicopter’s resting spot near Mt. Julian within a few sols. The flight was performed exactly as planned, with Ingenuity losing radio link at approximately 8 meters (matching our telecom models). Still, the lack of successful landing confirmation meant that the downlinked telemetry for this flight was somewhat less reassuring than typical post-flight downlinks. The team settled in for what was supposed to be a brief but suspenseful wait for Perseverance to catch up and provide confirmation that the intrepid helicopter had landed safely.

Roughly a week later, the rover began moving, but instead of heading southeast toward Mt. Julian as planned, the science team had decided to perform an investigation of the area around Powell Peak. The Ingenuity team members, thankful for the unplanned break but still worrying about the unknown state of the helicopter, prepared for another week or two of waiting. Weeks soon turned into months as the rover overcame various challenges to its schedule. The poor structural integrity of the rock in the area foiled two sampling attempts before the rover team was able to successfully capture a sample on Sol 822 and successfully seal it 10 sols later. People following the rover’s activities will know that this campaign proved to be far more challenging than anyone imagined. Finally, after another week of activities, Perseverance was close enough to re-establish communications with Ingenuity on Sol 837.

In total, Ingenuity had been out of contact for 61 sols, an eternity when the outcome of the flight was unknown to the team. This marked the longest period of helicopter inactivity since Perseverance and Ingenuity landed on the planet. During this time, the Ingenuity team had largely deactivated, with members reallocating their time to work on other projects.

As the team dusted off the cobwebs and started bringing down log files and images from the flight, it became apparent that the helicopter had spent the last two months parked on something truly remarkable. Sitting directly under Ingenuity’s feet, spread over the fractured rock of the riverbed, was a collection of cobbles and pebbles unlike any that scientists had seen before. Many were partially eroded and exhibited a vesicular texture more reminiscent of fresh basalt. These rocks immediately garnered a powerful reaction from project scientists, who requested that Ingenuity perform a dedicated science scouting flight as soon as possible.

The Science Flight That Wasn’t

The team jumped at the rare opportunity to provide valuable and exciting advanced science reconnaissance, but the flight would eventually pan out in a very different way. Flight 53 was to be an extremely interesting flight with extensive RTE (color camera) imagery at low altitude to return a plethora high-resolution ground scans covering portions of the riverbed slightly north of the original landing location. Partway through the flight on Sol 864, however, a synchronization issue with the time-critical navigation camera triggered the “LAND-NOW” fault protection routine in the guidance navigation and control (GNC) subsystem. As the name implies, this caused Ingenuity to abort the flight and land immediately where it was.

Ingenuity corrects its spatial orientation estimate by tracking the movement of ground features within these navcam images, but these data must be perfectly time-synchronized with measurements from the inertial guidance system to provide valid corrections. In the case of Flight 53, this synchronization step mysteriously failed in a way that hadn’t been observed on any of the prior 52 flights on Mars or during the years of ground testing that preceded them. As of this writing, the precise cause is strongly suspected but has not been conclusively proven.

During the development phase of the mission, many fault responses were considered, but with a system as inherently unstable and time sensitive as a helicopter, the best response is almost always to land as soon as possible. Following the R8.0 software update in October of 2022, the fault response had been subtly changed. R8.0 included a new and valuable hazard-divert capability, allowing the helicopter to intelligently shift its landing target to avoid areas that appeared dangerous to its cameras. This update also applied the hazard-divert behavior to the LAND-NOW response. In what was surely a fortunate twist of fate, the GNC team had discovered a peculiar and potentially fatal interaction between these two behaviors just one month prior to flight. By the time of Flight 53, operational changes had been put in place mitigate the issue (preventing the use of hazard-diverts during emergency landings). The Flight 53 LAND-NOW executed precisely as designed, getting the helicopter on the ground quickly and safely. This event was unprecedented and is Ingenuity’s first emergency landing on Mars.

By the time the team had assessed the issue, Perseverance had caught up to the helicopter and passed it on Sol 871, removing the need to complete the imaging planned for Flight 53. Ingenuity returned to the skies of Mars with a short pop-up Flight 54 to get a fix on its location and then resumed its scouting duties at a new location with Flight 55 on Sol 881.

Parting Thoughts

The team is in a constant battle with minimizing and balancing various risks. Ingenuity was flown out of comms range to guard against the very likely possibility that project scientists would opt to head west immediately. This illustrates the difficulty in making effective plans when the ability to execute those plans is entirely dependent on another vehicle, itself subject to unexpected events. This is doubly true since the science mandate of Perseverance rightly dictates that its plans should change as new discoveries are made and new data become available. Even without these inherent coordination challenges, sometimes unexpected events can derail even the most well-laid plans as they did in Flight 53. No one on the project could have predicted the eventual outcome that resulted in the loss of two months of helicopter mission time and the unfortunate loss of one of the most exciting scouting flights in recent memory. Nevertheless, the helicopter will continue up the ancient river delta, balancing risks and providing scouting for the rover where possible. The two Mars vehicles will soon reach an area where the rover is scheduled to loiter for several months. This should significantly relax planning constraints and provide an opportunity for the helicopter to engage in a wider variety of activities.

Written by Travis Brown, Chief Engineer Ingenuity Mars Helicopter at NASA's Jet Propulsion Laboratory