As detailed in our last blog post, for the first time in our yearlong extended mission we had a loss of communications with Ingenuity from the downlink of May 3 (Sol 427) and May 4 (Sol 428). After a week of anomaly investigation, two sols dedicated to data collection, and the heroic efforts of the Perseverance and Ingenuity operations teams, I am very happy to report that we have reestablished reliable communications with Ingenuity. Based on all available telemetry, the helicopter appears healthy, and we have resumed a modified form of operations. Assuming winter recommissioning activities complete nominally, Ingenuity’s 29th flight may occur in the next few sols.
Telemetry from Ingenuity confirmed that the loss of communications was due to insufficient battery state-of-charge (SOC) going into the night, which resulted in a reset of our mission clock. This daily SOC deficit is likely to persist for the duration of Martian winter (until September/October).
Challenges like these are to be expected: After hundreds of sols and dozens of flights beyond the five flights originally planned, the solar-powered helicopter is in uncharted terrain. We are now operating far outside our original design limits. Historically, Mars is very challenging for spacecraft (particularly solar-powered spacecraft). Each sol could be Ingenuity’s last.
Below is a summary of our anomaly investigation over the last two weeks, our elevated winter risk posture, and the possible paths ahead for Ingenuity during winter and beyond.
Sol 427 and 428 Loss of Communications
We have reached the point in Martian late fall/early winter at which Ingenuity can no longer support the energy demands of nominal operations. Starting on the evening of Sol 426, we believe Ingenuity started experiencing overnight battery brownouts (drops in the battery’s voltage), which reset the electronics. Due to the seasonal decrease in available solar energy, increases in airborne dust density, and the drop in temperatures, the energy demand to keep the electronics powered and warm throughout the night has surpassed Ingenuity’s available energy budget.
Below is a plot of the environmental conditions at Jezero Crater. The black line is a climatological model of airborne dust density (called “tau”) over a Martian year, and the expected daily insolation (amount of Sun’s rays reaching the solar panel) illustrated in the green dotted line. We are now in the part of the Martian year with peak tau, combined with falling insolation. Not shown in this plot is the additional trend of lower average daily temperatures, which also adds to Ingenuity’s energy demand. We expect to be in this challenging winter energy paradigm until around Sol 600, at which point we expect to return to being power-positive from sol to sol.
While in winter, each night we expect our battery SOC will fall low enough to where:
- Cold Cycling Components: Our heater thermostat can no longer keep the battery (and surrounding electronics) at their programmed setpoint (e.g. minus 13° F / minus 25 ° C). As a result, we expect that Ingenuity’s electronics are reaching overnight ambient temperatures of approximately minus 80 °C (minus 112 °F).
Impact 1: This cold cycling each night presents a lifetime risk to our electronic components.
- Cleared Mission Clock: The battery can no longer support our voltage regulators and mission clock. This nightly brownout resets (or clears) our mission clock.
Impact 2: Each morning when Ingenuity warms up, charges its drained battery, and attempts to turn on its computer, it is booting according to a misaligned alarm schedule, which is out of synchronization with Perseverance.
When Perseverance attempted communications with Ingenuity on the morning of sols 427 and 428, Ingenuity did not respond because it was waking up at the wrong time. The team reacted by uplinking a full-sol search activity with the Helicopter Base Station (HBS) onboard Perseverance to attempt to communicate with Ingenuity during one of its misaligned alarm widows, which succeeded.
Since Sol 429 and every sol since (with the exception of Sols 444 and 445, which did not contain helicopter activities), we have been in daily contact with Ingenuity by using similar morning search activities during what we believe to be the most likely times when Ingenuity would be sufficiently charged to attempt booting its electronics. These morning search activities reprogram the helicopter’s mission clock each sol, which, for the duration of that sol, enables additional scheduled activities to make use of the energy that we do have available. Currently, we reach sunset with ~68% SOC, with an estimated need of at least 70% to keep everything powered overnight. Our 2% SOC deficit is expected to grow to a 7% deficit once we reach winter solstice (Sol 500 in July), at which point conditions will start to improve.
All telemetry downlinked so far suggests that Ingenuity is healthy, with no signs of damage from the overnight cold cycles. This morning-search followed by evening activities is our new normal for the immediate future.
Winter Risk Posture
Our latest models suggest that regardless of modifications to our overnight thermostat strategy, it will be extremely challenging or even impossible to keep our electronics core module (ECM) components warm and within their nominal temperatures overnight. Although component failure has always been a risk that we have carried since rover deployment, that risk is now magnified. ECM components were normally kept warm overnight by our battery heater (typically set to 5°, minus 4°, or minus 13°F, or minus 15°, minus 20°, or minus 25°C). We expect that ECM components are now thermal-cycling down to the ambient overnight temperatures of minus 112 °F (minus 80 °C). We do have limited ECM component testing to suggest that select components may survive through the winter, but we cannot predict how the entire ECM will fare throughout winter. Cold-soaking electronics is believed to have caused the end of the Opportunity and Spirit Mars rover missions.
Winter Operations and Beyond
Given our elevated risk posture, our focus in the last several sols has been to prioritize data downlink from Ingenuity to the HBS. We have a handful of Heli-to-HBS transfer activities left before all unique data are copied from Ingenuity to the HBS. Specifically, we are copying flight performance logs, electronics logs, and high-resolution color images from the last eight flights that are still onboard Ingenuity.
After all critical logs are transferred, the team will proceed with a recommissioning phase during which we will reestablish Ingenuity’s flight-readiness given our ongoing overnight cold-cycling. Like during the technology demonstration phase, we will perform a high-speed spin before proceeding to flight. Should Ingenuity receive a clean bill of health, we would be ready to execute a short sortie to the southwest in Flight 29. This flight will improve our radio link for approximately the next four to six months while Perseverance samples at the river delta.
In the meantime, the Ingenuity flight software team will be preparing a series of upgrades to enable advanced navigation features. These new capabilities will help Ingenuity ascend the river delta and continue its missions as a forward scout for Perseverance past winter.
The Perseverance and Ingenuity operations teams have done an extraordinary job in reestablishing reliable communications with Ingenuity. As of Sol 446, the team has transferred ~3,500 megabits (Mb) of data from Ingenuity to the HBS on Perseverance. There are approximately ~400 Mb of data remaining on Ingenuity of noncritical logs and images that will be transferred opportunistically, while the team forges ahead with recommissioning efforts this week. If the telecom link between Perseverance and Ingenuity remains stable, and Ingenuity receives a clean bill of health, we hope to execute Flight 29 in the next several sols.