By Grace Peng
Software, and lots of it!
The issue of Toyota's sudden acceleration problem came up at both the last PTA meeting I attended and at my mother-in-law's dinner table last weekend. Perhaps this digression about real-time software is of general interest.
Software has become embedded into so many things we use every day that it has become invisible to us. Because of its ubiquity, we are not using independent pieces of software, but rather systems made up of smaller interacting subsystems, each with their own software. Because of this complexity, it can be difficult to trace a root cause when a problem arises.
In grad school, I wrote and worked with computer models that try to describe physical processes. That led to numerical weather prediction (NWP), which led to satellite data processing -- how the signal off a satellite sensor gets turned into bits which flow down to a groundstation and get turned into information that helps NWP do a better job of predicting the weather.
That's all software. But there is a fundamental difference between the type of software I have written and real-time systems.
No one cares how long it takes for a grad student's model to run, except for the grad student that wants to finish up and graduate.
NWP predictions have to run faster than elapsed model time or else it would be a hindcast instead of a forecast.
Real-time software must run at the same speed as elapsed time. When they are inseparably installed in a machine to control or take readings from it, they are called real-time embedded software. So the answer to the title question should have been that both automobiles and spacecraft contain large amounts of embedded real-time software.
The similarities don't stop there. Both are complex systems of subsystems, each with their own computer and software. The manufacturer of the automobile or spacecraft likely did not build all of the subsystems. So they become an integrator of subsystems and software. The system becomes very complex, and difficult to test robustly. That is, it is difficult to design tests that cover all possible scenarios or paths down a decision tree.
In the satellite software world, the standard is to "test like we fly." (I imagine the automotive industry does the same.) All of the software in the system will be tested together for weeks or months at a time, with all manner of monkey wrenches thrown in the mix, to make sure everything behaves in the intended manner.
Toyota's engineers didn't find a sudden acceleration problem before the cars went to market. Even after the accidents, they tried to simulate the events the motorists described and they still couldn't find a software root cause. What were they missing?
When you can't find what you are looking for, it is logical to ask someone with fresh eyes to look for it. You can't hand over your software to a competitor to test. That's too much temptation. It's better to ask a neutral party, perhaps someone in government for help. In case there is an industry-wide myopia to the root cause, it helps to ask someone in a different industry who does the same functional task.
Hence, NASA Goddard Space Flight Center was called in. GSFC integrates many satellites and has both the laboratory capability and the human expertise to perform that kind of testing and analysis.
[Speaking as a private individual and not as a representative of my employer or any of the government agencies that fund us, I would like to make an appeal. Those scientists, engineers and technicians need to get paid so they don't drift into other industries that do pay. If they weren't there, who would be there for us when a problem arises? It pains me when our politicians spout off about "killing the beast" and "getting government off the backs of industry." I think we should send them on a fact-finding mission to Somalia so they can see what a country without a functioning government looks like. Perhaps they want to move there?]
Remember the part about asking fresh eyes for help when you are stuck? A group of similar individuals can fall prey to groupthink. How can we trust a piece of software or a new drug with our lives if we aren't sure about the design and testing process? And how can we do that if only a narrow cross-section of society engages in that process? That's the best argument for a diverse workforce I have ever seen.
In science and technology, we test and test again. The more the results stand up to multiple experiments in different laboratories, and different experimental designs, the more we can trust the results. STEM (Science/Technology/Engineering/Mathematics) needs people with all kinds of backgrounds to point out things that other people may have missed.
In closing, I would like to point out it is never too late to learn something new and learning come can come from unexpected places.
My daughter is on her middle school Lego robotics team. Like many public school teams, they are at a disadvantage relative to the boy/girl scout troops and private schools because they have one teacher to 32 students. A teammate's dad had been helping out one afternoon a week. When his project at his market work was heading into PDR*, he sent an e-mail asking me to take over.
I duly bought a Lego NXT 2.0 kit and some books, installed the software on my laptop and set out to learn how to program a Lego robot. It turns out that the Lego API (application programming interface) is a variation of LabVIEW, a real-time programming tool used in many labs. At the 2010 Workshop on Spacecraft Flight Software, I learned that LabVIEW is used by MIT students to prototype spacecraft control algorithms!
My daughter helped me program a robot to run a (American football) post play. That means I do have some real-time programming experience after all. But I had to learn it from my 10-year-old.
I would like to thank Keith Blount for his excellent introduction in DIY software. Let's turn every house into a software house.
*PDR stands for preliminary design review. All satellite programs go through thorough design reviews at various stages or milestones. Engineers and managers for both the contractor and the buyer work very long hours leading up to and during the design review process. PDR is the last chance to catch gotchas before proceeding to build the satellite.
I don't mean to shill for my blog, but I have covered this beat for years.
Mommy Art (and Science) explains why software jobs can be highly compatible with child-rearing.
Rockin' deals with the historical reason why the NWP field enjoys a relative abundance of women. I collected one oral history about what happened before, during the depression and WWII, but I need to collect more evidence. Better yet, a professional historian should cover this because I need to go to work now.
Grace Peng
has a day job at the intersection of science, technology and
governance/policy. She also does a split shift at home as a mother and
wife to a field scientist. She blogs about science, the culture of making, and work/life issues at Bad Mom, Good Mom.
