When the Error Message Comes From 50 Million Miles Away

…spacecraft engineers have to get creative.

drifting galileo
The Galileo spacecraft was already far from Earth when problems cropped up with its antenna. NASA

Robotic spacecraft have traveled the solar system for more than 50 years, exploring places astronauts can still only dream of visiting. But with no human aboard, when something goes wrong, it’s up to ground control to pinpoint the problem—and fix it from millions of miles away. Sometimes all the ingenious engineering in the solar system wasn’t enough to keep a mission from being lost in space.

In a small, windowless room at Florida’s Kennedy Space Center on October 18, 1989, NASA project manager Bill O’Neil was monitoring the launch of space shuttle Atlantis. He was close enough to feel the vibrations but he focused on the special cargo being carried aloft: the Galileo spacecraft, finally on its way to Jupiter and its moons. Astronauts took a picture of the spacecraft (above) shortly after it left the cargo bay of Atlantis.

“I was practically numb,” O’Neil says, describing that day. Galileo had been delayed many times since 1982; it was even packed up at Kennedy, ready for a May 1986 ride into orbit, when Challenger exploded that January. New safety rules in the wake of the accident forced the team to replace the hydrogen-fueled Centaur upper stage that would rocket Galileo toward Jupiter in a speedy two-year journey with a safer but less powerful solid-fuel upper stage and a new trajectory—one that would take six years and require slingshotting around Venus and, twice, Earth for gravity assists.

For a year and a half, Galileo sailed smoothly through the inner solar system. On April 11, 1991, the operations team sent a command to deploy the mission’s most important piece of communications hardware: the high-gain antenna, a towering umbrella with a powerful transmitter. Within minutes, they knew there was a problem: The antenna had failed to deploy.

Three of the extension ribs in the umbrella were stuck. For the next year, the Galileo team worked to troubleshoot the spacecraft. They sent commands to alternately turn the vehicle away from its sun orientation, then back again, hoping the heating and cooling would expand and contract the metal in the antenna tower and “walk” the ribs out of their lockdown. They tried “hammering” the deployment motors by pulsing them repeatedly. Nothing worked. “At first there was a lot of optimism that we would figure it out and get the [antenna] open,” says O’Neil. “That turned out not to be true.”

Fortunately, Galileo itself had a fix. Compared to its high-gain companion, the spacecraft’s low-gain antenna was 10,000 times slower in data transmission, but it could have been worse. The spacecraft’s engineers made radical changes to both its software and its ground control receivers, increasing the low-gain antenna’s initial transmission rate by 100 times.

On December 7, 1995, Galileo reached Jupiter and became the first spacecraft to settle into the giant planet’s orbit. “We had undertaken an audacious tour of Jupiter’s miniature solar system, and despite all the challenges, we never missed an encounter,” O’Neil says. The signals may have been weak, but NASA received them—including evidence suggesting that under the surface of the moon Europa, there are liquid saltwater oceans—until Galileo was finally deorbited in 2003.

See the gallery below for more stories of dramatic spacecraft saves.

The Curiosity Rover

(NASA / JPL-CalTech)

The Curiosity Rover, NASA’s Mars Science Laboratory, has been grabbing attention since August 2012, when it landed in Gale Crater in a spectacular fashion. The rover began its trek toward Mount Sharp, about five miles away, and almost immediately began sending back discoveries from the Martian surface. Then one night last February, project mission manager Jim Erickson was awakened by a call. “Ground control said they were getting some interesting telemetry from the Mars rover relay,” Erickson says.

Team members at NASA’s Jet Propulsion Laboratory in California had noticed the problem when Curiosity failed to send back the data it had recorded for the day and go into regular sleep mode. That meant something was wrong with the rover’s main computer. The engineers woke up Erickson and switched the rover over to its secondary computer, which they call the B-side, a move that automatically puts the rover into safe mode so the team can root around for the problem.

Erickson was concerned, but he was a veteran at space hardware troubleshooting, having worked on the Mars rovers Spirit and Opportunity—and Galileo. Sure enough, his team located damaged memory hardware in the A-side primary computer. The team could reprogram the A-side to not recognize the damaged parts, but Curiosity was working just fine using the B-side, so they decided to keep it as the rover’s primary system. In the space business, it’s always good to fly with a spare.

Hayabusa

(NASA )

From the very beginning, Hayabusa was plagued with problems. The spacecraft, launched by the Japan Aerospace Agency (JAXA) in May 2003, had a bold mission: Rendezvous with the asteroid Itokawa, 200 million miles away, collect a piece of it, and bring it to Earth.

Just a few months into its two-year voyage to Itokawa, a solar flare hit Hayabusa, severely damaging its solar batteries. The reduction in electricity hindered the efficiency of the spacecraft’s ion engines, delaying arrival by three months. Hayabusa’s relatively small target was racing through space at 57,000 mph, and since the spacecraft had to leave Itokawa by November 2005 to be on the proper orbital path to reach Earth, the delay forced engineers to recalculate the probe’s trajectory and shorten its stay.

Shortly before the probe arrived at Itokawa, two of its four reaction wheels failed. Loss of these fuel-conserving attitude adjusters forced the team to make creative use of the chemical thrusters to maintain its orientation. “There were some long days and a few sleepless nights wondering if the spacecraft would survive,” says Don Yeomans, a scientist at the Jet Propulsion Laboratory and member of the Hayabusa team.

By September 2005, Hayabusa was on final approach to Itokawa. The plan was for the spacecraft to release its grasshopper-size mini-lander, MINERVA, to collect data and images from the surface. The main spacecraft would then approach and bump the asteroid to kick up and capture particle samples. But because of the communications delay due to Hayabusa’s distance from ground control, the command to release MINERVA arrived just after the main spacecraft’s automatic altimeter fired its thrusters to keep the craft away from the asteroid. Result: It was too high for MINERVA’s launch. The tiny lander drifted off into space. (contd.)

Hayabusa's Remains

(NASA)

(contd.) That was not the end of the mission’s woes. Instead of “bumping” the asteroid, Hayabusa strangely hovered about 30 feet above it, until scientists sent a command for it to abort so they could try again. Later they realized the spacecraft hadn’t been hovering—it had landed, but failed to fire the two “bullets” designed to chip flakes into a sample collector. Project scientists were hopeful that enough dust had been kicked up to take a sample, so they sealed the container, but attempted one more try at the maneuver. The bullets failed to fire again, and then Hayabusa’s thrusters sprung a leak. The spacecraft went into safe mode, and as it drifted away from Itokawa, all communication was lost.

The team found Hayabusa two months later, as a small blip on a radio-wave screen at the Usuda Deep Space Research Center in Saku City, Japan. It took another four months to fully recover control and reestablish communications. As Hayabusa limped toward Earth powered only by the weakened ion engines, its arrival date was pushed from 2007 to 2010. Some doubted that the spacecraft could make it at all.

“But the Hayabusa spacecraft and the JAXA-led team answered every challenge,” says Yeomans. The spacecraft burned up over southern Australia on June 13, 2010, and the sample container capsule floated intact to the ground as planned and was recovered by scientists (above). Inside the container was about 1,500 grains of dust. In spite of all the near catastrophes, the mission became the first to return samples from an asteroid, providing scientists valuable first-hand study of these solar system objects.

SOHO

(ESA)

Sending a machine out into space requires taking many, often mysterious, risks, so preventable human error makes a failure that much more painful. In June 1998, Bernhard Fleck and his wife had just moved near NASA’s Goddard Space Flight Center in Maryland to work on the Solar and Heliospheric Observatory, or SOHO, which had already completed its primary two-year mission: studying the structure and changes of the sun. Fleck, a project scientist with the European Space Agency, which had joined NASA on the solar probe, was unpacking boxes when he got a call from his team. During a routine maneuver conducted every three months, SOHO lost its attitude lock on the sun. The loss triggered a safe mode called Emergency Sun Reacquisition, or ESR, which fires the spacecraft’s thrusters in an attempt to get the craft back facing the sun.

“At first I wasn’t worried,” Fleck says. “I knew it would be a long night, but SOHO had gone into ESR before. We were bringing it back, and had a second ESR. Then a third. After five minutes someone in the control room said, ‘Oh shit! We made a mistake!’ It was our fault.”

Recovering SOHO from safe mode would have gone smoothly the first time if it weren’t for two crucial errors. A pre-programmed command code made it so one of the gyroscopes, which was supposed to activate during ESR to get the spacecraft realigned with the sun, failed to turn on. A second pre-programmed code was sending the team incorrect readings from another gyroscope. To make matters worse, the team had rushed to get SOHO out of safe mode instead of analyzing the problem, during which time they might have found the command errors. With the thrusters firing but the gyroscopes uncontrollable, SOHO was left spinning wildly, unable to collect solar energy. Eventually, the spacecraft switched off, terminating the communication link. One can imagine a shrug and maybe a robotic wish that the humans would get it together.

“The feeling was very grim,” Fleck says, “But the team did not give up.” The first “long night” turned into long months searching the skies to find the spacecraft, and Fleck updated a website every day on the recovery process. Then came the update on August 4: “The search was finally successful!” Using a combination of the Arecibo Radio Telescope in Puerto Rico and NASA’s Deep Space Network, the team found SOHO floating out in space and reestablished communications.

It was a few more months before the team could return SOHO to full operations, slowly charging its solar batteries and defrosting its hydrazine fuel. At any time, the tank or pipes could have ruptured. As the spacecraft emerged from its icy state, the team was thrilled to find that all 12 science instruments were still operational. The gyroscopes, however, could not be salvaged, so Fleck’s team improvised a unique stabilizing system using the spacecraft’s reaction wheels, a type of flywheel.

Seventeen years after launch, SOHO continues to make significant contributions to our understanding of the sun. “It was a blunder with a very lucky outcome,” Fleck says. “Though it was a nightmare, I learned you do everything you can, and you can do what you think is impossible.”

Deep Space 1

(NASA JPL)

In 1995, NASA kicked off the New Millennium Program, an initiative to test technology in space to benefit future space science research. The first platform was Deep Space 1, which launched in October 1998 with innovative systems like autonomous navigation, miniaturized cameras, and an ion propulsion engine that needed to prove its usefulness for long-duration missions.

The primary 11-month technology demonstration went well, according to NASA project manager Marc Rayman. Then the mission team sent Deep Space 1 off on a two-year mission to fly by the comet Borrelly (pictured) in September 2001. But just two months into the trip, the spacecraft’s star tracker, a critical space navigation tool, failed. The initial reaction from NASA officials was that the incident was fatal to Deep Space 1. “But when you have a team of passionate and ambitious explorers,” Rayman says, “you don’t give up easily.”

Indeed, the team saw the situation as an entirely new test for their suite of equipment, and dreamed up ways to jury-rig the systems to replace the star tracker. After seven months, engineers had developed new command software that would make use of the miniature camera and the autonomous navigation system—transmitting the software update to the spacecraft, now a couple of hundred million miles away, at a painfully slow rate. At one point, the data link to the spacecraft was interrupted, causing everything that had been uploaded up to that time to be lost. That was the lowest point for Rayman and his team, he says. Still, they started again.

On September 21, 2001, Deep Space 1 flew into the icy dust cloud of Borrelly and took the first images of the nucleus of a comet.

Freelance writer Zoe Krasney lives in New Mexico, surrounded by the history and future of aviation and space exploration.