NASA Jet Propulsion Laboratory California Institute of Technology JPL HOME EARTH SOLAR SYSTEM STARS & GALAXIES SCIENCE & TECHNOLOGY JPL Email News RSS Mobile Video
Follow this link to skip to the main content
JPL banner - links to JPL and CalTech
left nav graphic Overview Science Technology The Mission People Spotlights Events Multimedia All Mars
Mars for Kids
Mars for Students
Mars for Educators
Mars for Press
+ Mars Home
+ Rovers Home
features
list features
Masterminds of Scientific Art and Rover Movement on Mars

July 09, 2004

Opportunity's panoramic camera image consists of 225 individual frames, which are knit seamlessly together by behind-the-scenes masterminds.
Opportunity's panoramic camera image consists of 225 individual frames, which are knit seamlessly together by behind-the-scenes masterminds."
Image credit: NASA/JPL/Cornell
Link to Full Res
Imaging Masterminds Give Human-Scale View of Mars

The world gets to see the rusty, dusty martian terrain as if humans themselves were riding atop the rovers. Who's responsible for our front-row seats on Mars? The masterminds working tirelessly behind the scenes in the Multimission Image Processing Lab (MIPL) at the Jet Propulsion Laboratory.


XYZ-axis terrain map for Opportunity, with blue contour lines representing elevation, so rover drivers can see the slope going down into Endurance Crater.
XYZ-axis terrain map for Opportunity, with blue contour lines representing elevation, so rover drivers can see the slope going down into Endurance Crater.
Image credit: NASA/JPL
Link to Full Res
Like professional basketball stars, MIPL (pronounced ‘mipple') team members at Jet Propulsion Laboratory perform amazing technical feats and shoot images from Mars into the Internet in record, near real-time before their shot clocks expire many times a day.

Not only do they create the seamless landscape pictures from Mars, but the imaging team also keeps the science game moving by creating image products that enable the rovers to rove and the scientists to collect science.

"We're what's under the hood," explained Helen Mortensen, former MIPL team leader and development engineer. "You won't see us at press conferences, but we're behind the creation of the images and the safe movement of the rovers."


First image of the moon taken by Ranger 7 on July 31, 1964.
First image of the moon taken by Ranger 7 on July 31, 1964.
Image credit: NASA/JPL
Link to Full Res
The Invention of Digital Cameras and Image Processing

"For digital imagery, MIPL is where it all started in the 1960s," explained Payam Zamani, MIPL design and system engineer.

In the early days of space exploration, engineers had to figure out a way to take a basic television signal and convert it to zeros and ones in order to store image information onto a computer and beam the pictures back to Earth.

Thus, digital cameras and digital image processing were invented at JPL for the space program, and enabled the first views of the moon and Mars.


On November 27, 1971, Mariner 9 (the first spacecraft to orbit another planet!) took this image of the ancient volcano, Olympus Mons, standing above a martian dust storm.
On November 27, 1971, Mariner 9 (the first spacecraft to orbit another planet!) took this image of the ancient volcano, Olympus Mons, standing above a martian dust storm.
Image credit: NASA/JPL
Link to Full Res
We've certainly come a long way in a few decades. Not only do we have better and better pictures of Mars and the other planets we're interested in exploring, but we also have seen a number of benefits here on Earth. MIPL is an unsung hero of sorts.

"The technology designed for the space program has now made it into the commercial world, and anyone who uses a digital camera and a home computer to download and process pictures is using MIPL technology," said Zamani.


Helen Mortensen, former MIPL team leader and development engineer, in JPL's imaging lab.
Helen Mortensen, former MIPL team leader and development engineer, in JPL's imaging lab.
Image credit: NASA/JPL
Link to Full Res
Scissors, Paper, and Paste for Space-Age Images?

"When the first photos were transmitted from space, scientists would have to use scissors to cut and paste multiple images together to make a panorama, and it would take days or months to build a picture of a large landscape from another planet," said Mortensen.

Now, utilizing software that JPL has been developing and improving for over 30 years, scientists and engineers can see images from Mars within 60 seconds from the time the data hits Earth.


Science team members Eric DeJong and Zoe Learner participating in a Mars event at the St. Louis Science Center.
Science team members Eric DeJong and Zoe Learner participating in a Mars event at the St. Louis Science Center.
Image credit: NASA/JPL
Link to Full Res
MIPL processes 30 to 250 images a day, per rover. Through a MIPL software program named File Exchange Interface, museum partners around the nation can also get high-resolution images from Mars out to the public in nearly real time.

More than 100 museums were able to participate, bringing the excitement of Mars exploration to people in the communities where they live. MIPL team members were even able to visit some of the museums, talking with students about their work.


3D stereo image of the rover's arm, taken by Opportunity's front hazard-avoidance cameras.
3D stereo image of the rover's arm, taken by Opportunity's front hazard-avoidance cameras.
Image credit: NASA/JPL
Link to Full Res
Three-Dimensional Terrain Maps and Reaching the Rocks

A picture of a beautiful landscape speaks a thousand words, and shows what MIPL team members pull off on a daily basis.

However, other important but less obvious products generated by MIPL are the behind-the-scenes, high-tech stereo images that are used by rover drivers to maneuver safely on Mars.

Without knowing the distance to each rock near the rover or slope in the terrain around the rover, rover drivers could not safely command the rover to drive toward a specific location or send the robotic arm out to reach a target with the scientific instruments.


Payam Zamani, MIPL design and system engineer.
Payam Zamani, MIPL design and system engineer.
Image credit: NASA/JPL
Link to Full Res
MIPL image processing uses some basic geometry to find distances of objects from the rovers.

"Geometry was my favorite subject in high school, but back then I didn't know why those techniques were taught, and now it's good to see that stuff teachers tell you to do actually can be used on exciting things like space missions," said Zamani.

Working just like human eyes, the MIPL team uses two images taken from different angles -- a left and a right -- to determine depth in the images from Mars.

"If you hold your finger in front of you and close one eye, you'll see that your finger moves if you look at it using your left eye or your right eye. That's parallax.

Now move your finger out far from your eyes and look at it from your left eye versus your right. Things that are closer jump more than things that are far away," explained Zamani.

The MIPL team finds common features in every left and right image pair taken from the same rover position, so they can calculate a triangle and figure out the distance of the common object from the rover, making a three-dimensional map.


Left-eye half of a stereo pair of mosaics from Spirit's navigation camera, June 8, 2004.
Left-eye half of a stereo pair of mosaics from Spirit's navigation camera, June 8, 2004.
Image credit: NASA/JPL
Link to Full Res
Right-eye half of a stereo pair of mosaics from Spirit's navigation camera, June 8, 2004.
Right-eye half of a stereo pair of mosaics from Spirit's navigation camera, June 8, 2004.
Image credit: NASA/JPL
Link to Full Res


This stereo mosaic was created from left and right navigation camera images acquired on June 8, 2004.
This stereo mosaic was created from left and right navigation camera images acquired on June 8, 2004.
Image credit: NASA/JPL
Link to Full Res
Astoundingly, MIPL software does that pixel by pixel for 1,000,000 pixels per picture pair.

"There are very heavy-duty math algorithms in the MIPL correlation software that find matching pixels in the right image for every pixel in the left image," said Bob Deen, chief software developer for the MIPL Mars team.


Range map created by the MIPL team to enable rover drivers to command Spirit and Opportunity to travel the correct distance to a target.
Range map created by the MIPL team to enable rover drivers to command Spirit and Opportunity to travel the correct distance to a target.
Link to Full Res
Know Your X,Y, Zs to Move on Mars

MIPL's 3D maps enable the rover drivers to command Spirit and Opportunity from over 100 million miles away.

The IDD (Instrument Deployment Device) maps and range maps determine which points are reachable by the arm's various instruments.

The XYZ maps and terrain wedges represent the 3D terrain to ensure the rover won't hit anything, and the slope images show slope changes that could be hazardous to drive down or up, Deen explained.


Bob Deen, chief software developer for the MIPL Mars team, is also known as the 'Mosaic Czar.'
Bob Deen, chief software developer for the MIPL Mars team, is also known as the "Mosaic Czar."
Image credit: NASA/JPL
Link to Full Res
The MIPL team creates these maps from telemetry that comes down from each rover multiple times a day.

They organize the "metadata" for every image, which is over three hundred lines of essential information per image, such as exposure time, temperature, and rover body orientation and articulation at the moment the image was taken.

The team must process all of this information and whip out 14 vital products per image at a breakneck speed.


Reachability map created by the MIPL team to determine which points are reachable by the robotic arm.  Colors represent what areas are reachable by the various science instruments.
Reachability map created by the MIPL team to determine which points are reachable by the robotic arm. Colors represent what areas are reachable by the various science instruments.
Image credit: NASA/JPL
Link to Full Res
That way, scientists and engineers have enough time to digest the information and utilize the maps to command the rover in time to meet the next communications deadline each day.

In order to expedite the map-making and mosaic-making process, MIPL created a data "pipeline."

Organizing Millions of Bits of Information

Within 60 seconds of receiving data from Spirit and Opportunity, MIPL team members dribble and shoot thousands of bits of information into a "pipeline."

The pipeline processes all of the data from Mars into an automated assembly line that enables scientists, engineers, and the public to see and comprehend the martian world in all of its glory.


Doug Alexander, MIPL team leader and development engineer working in the 'bullpen' at JPL.
Doug Alexander, MIPL team leader and development engineer working in the "bullpen" at JPL.
Image credit: NASA/JPL
Link to Full Res
A pipeline pilot ensures that all the information flying into the system is heading to the right destination.

"The MIPL pipeline is like a gourmet catering service that preps, cooks, and serves packets of information from Mars into the right people's hands on Earth almost instantaneously," said Doug Alexander, MIPL team leader and development engineer.

The pipeline is the "smarts" and "big glue" that runs a lot of programs, Zamani explained. "It says, ‘If I have a left pancam image, I need a right pancam image to make a 3D image.'"


Three-dimensional slope map that helps rover drivers create safe routes for Opportunity to maneuver through steep terrain inside Endurance Crater.
Three-dimensional slope map that helps rover drivers create safe routes for Opportunity to maneuver through steep terrain inside Endurance Crater.
Link to Full Res
Depending on rover power levels and the amount of time Spirit and Opportunity have to transmit data via the Odyssey orbiter relay or a direct-to-Earth antenna link, team members on Earth receive only some of the data during communications downlinks.

The pipeline looks for any missing pieces of information based on time codes and knowledge of when information should have been collected. It may take four or five days to get all the images down to create one panoramic mosaic.

In the meantime, beyond getting the images for a broad landscape, the MIPL team must get enough pieces of information every day to create the 3D maps, so "the pilot watches the pipeline like a hawk," Zamani said.


MIPL team members at a recent cookout. (Front row, L to R) Steve Lavoie (husband of Sue Lavoie, does not work at JPL), Vadim Klochko, Carol Stanley, Oleg Pariser, Payam Zamani (Middle row, L to R):  Amy Chen, Sue Lavoie, Michael Cayanan, Elmain Martinez, Barbara McGuffie, Gary Yagi, Hyun Lee (Back row, L to R)  Doug Alexander, Charles Thompson (partially hidden behind Michael), Myche McAuley, Bob Deen, Helen Mortensen, Richard Norman, Paul Andres, Justin Maki
MIPL team members at a recent cookout. (Front row, L to R) Steve Lavoie (husband of Sue Lavoie, does not work at JPL), Vadim Klochko, Carol Stanley, Oleg Pariser, Payam Zamani (Middle row, L to R): Amy Chen, Sue Lavoie, Michael Cayanan, Elmain Martinez, Barbara McGuffie, Gary Yagi, Hyun Lee (Back row, L to R) Doug Alexander, Charles Thompson (partially hidden behind Michael), Myche McAuley, Bob Deen, Helen Mortensen, Richard Norman, Paul Andres, Justin Maki
Link to Full Res
Straddling Science and Operations

"The quality and volume of data from the rover mission is high, and the trick is to organize the information quickly and have a pipeline that never breaks," said Alexander.

The MIPL team developed methods to make sure that what the rover uplink team commanded the rover to send back to Earth did in fact come down the next day.

As the pipeline pilot does the checks and balances over tens of thousands of lines of data, other MIPL team members check the quality of the other 14 products. "We straddle the fence between science and operations," Alexander said.

Thrown into a small area called the "bullpen" for 12-hour shifts, six MIPL team members are generating the fundamentals that make the rover function around the clock.

The skilled professionals are constantly creating 3D terrain meshes and "gluing" together multiple pieces of data from the hazard-avoidance cameras (up-close images), navigation cameras (middle distance images), and panoramic cameras (far-away images) to give a different view of the martian landscape for multiple tactical purposes.

MIPL software products and team members even remove the fisheye view from camera images and create a human-eye view.

"The MIPL-engineered software has been optimized for processing rover camera data automatically, but the beautification of the mosaic products is a result of human intervention," Alexander said.


Original 'fisheye' view from Opportunity's front hazard-avoidance camera.
Original "fisheye" view from Opportunity's front hazard-avoidance camera.
Image credit: NASA/JPL
 
Link to Full Res
'Linearized' front hazard-avoidance camera image, in which MIPL
"Linearized" front hazard-avoidance camera image, in which MIPL "magic" straightens original fish-eye curves.
Image credit: NASA/JPL
Link to Full Res


Simple mosaic of panoramic camera images before the team fine-tunes the pieces of the puzzle.  Note how the rocks are doubled along the seams.
Simple mosaic of panoramic camera images before the team fine-tunes the pieces of the puzzle. Note how the rocks are doubled along the seams.
Image credit: NASA/JPL/Cornell
Link to Full Res
Non-stop Scientific Art

Creating the highly visible, aesthetically pleasing mosaics "involves manual editing and custom surface modeling," explained Doug.

The software developed at JPL can process camera data to quickly generate a mosaic, but detail-oriented, tireless team members must eliminate the doubling of the foreground rocks and match the brightness for one scene that may have been shot over multiple days.

Known as the Mosaic Czar, Bob Deen didn't take one day off for the first 90 days of the rover mission.


This 360-degree view of 'Bonneville Crater' was assembled from frames taken by Spirit's panoramic camera.
This 360-degree view of "Bonneville Crater" was assembled from frames taken by Spirit's panoramic camera.
Image credit: NASA/JPL/Cornell
Link to Full Res
"I was the general trouble-shooter at the beginning of the mission, and we all wanted to improve the software to generate products quicker for the rover drivers and press conferences, so it was too exciting to take a day off," said Deen. "The worst stress comes from the terrain meshes for the rover drivers because they can't plan the traverse until we combine the terrain information from various cameras."

Early in the mission, stress also flared when press conferences were happening every day. Scientists would often come to Deen and the MIPL team and request new mosaics or other special images at 5:00 a.m. for a press conference planning meeting at 6:00 a.m.

One night, after processing Mars images for 14 hours, Helen walked out of the office and noticed an odd iridescent glow in the distance. "I thought, ‘gosh, that's really strange -- I wonder what that is,'" said Helen. She later realized it was a band of the Sun rising on Earth . . . a sun she hadn't seen for a few days working the martian day.


This false-color image from Opportunity's panoramic camera shows a downward view from the rover as it sat on the edge of Endurance Crater, and is comprised of images taken from 10 rover positions.
This false-color image from Opportunity's panoramic camera shows a downward view from the rover as it sat on the edge of Endurance Crater, and is comprised of images taken from 10 rover positions.
Image credit: NASA/JPL/Cornell
Link to Full Res
Dedicated Team

No one on the MIPL team complains about the long hours, or the fact that they are the ones hidden behind the curtain.

"I have always liked science fiction and computers," Deen said. "My dad had a business doing tax returns in the early '70s on computers when computers were big mainframes in a room. I got an Apple II in high school and learned everything. I thought I wanted to work for Lucasfilm (of Star Wars fame), but when I found out in college that NASA hired people to run the computers for space science, I knew I wanted to do real science instead of science fiction. Now, after looking at all the rocks on Mars, I want to take a geology class," Deen said.


Students analyzing images from Mars with their mentor, Dr. Jeff Moersch of the Mars Exploration Rover science team.
Students analyzing images from Mars with their mentor, Dr. Jeff Moersch of the Mars Exploration Rover science team.
Image credit: NASA/JPL
Link to Full Res
"Our team has an age range of 21-year-old college students to people at the twilight of their retirement, and it's interesting to see how everyone gets along with a common goal," Alexander said.

The rover team is full of some of the brightest and most creative scientists and engineers who have to balance their various goals for their instruments.

"People work shoulder to shoulder with no envy, and they get over political work issues quickly in order to ensure the science gets accomplished," Alexander explained.

Everybody makes sacrifices depending on the most efficient tasks of the day and how the team can use the various instruments to help solve the mysteries of history of water on Mars. "It's amazing how much human energy is behind a six-wheeled rover," he said.


Athena Student Interns from Ramsay High School in Birmingham, Alabama check out 3D images from Mars while working at JPL with the rover science team.
Athena Student Interns from Ramsay High School in Birmingham, Alabama check out 3D images from Mars while working at JPL with the rover science team.
Link to Full Res
Rewriting Science Books

"When I was in grade school, I only saw blurry images of planets in our textbooks," Mortensen said. "Everybody is curious about the unknown, and one way to unveil that curiosity is to actually seek and give facts."

With the image processing technology MIPL has developed for NASA, students can use the Internet as their textbook and see crystal-clear images from Mars within hours of the moment the picture was taken on Mars.


Maestro
Computer screenshot of "Maestro," which is a free public version of the software that NASA scientists use to operate the rovers.
Link to Full Res
"For the missions I've worked on at NASA, including missions to Neptune and Jupiter, I have seen that technology actually goes into the public and ultimately benefits our entire society," she said. "Our current techniques for looking at data and analyzing the data are becoming faster and more sophisticated, and the commercial applications are infinite."

Working with every planetary mission at JPL, MIPL is creating new views of our universe and changing the way we see the solar system and beyond.


Laguna-Acoma high-school students from Casablanca, New Mexico research rocks on Mars with their teacher and science mentor at JPL.
Laguna-Acoma high-school students from Casablanca, New Mexico research rocks on Mars with their teacher and science mentor at JPL.
Link to Full Res
"My advice to any student who is interested in space is to learn as much math and science as you can. You are always going to see something really cool if you just keep your eyes open."

To learn about other star players on the MIPL team, please visit Zipcode Mars and type in the names of individuals in the team photo.


USA.gov
PRIVACY    |     FAQ    |     SITEMAP    |     CREDITS