Design in the Context of Use

In my career I have participated in projects that developed software in dramatically different ways. I've engaged in "throw it over the fence" (the old-school academic approach), "collaborative research" (one way to get funding from corporations in the early 90s), and now "empirical requirements analysis" in a "research station."

At Stanford University in the 1970s, the "Mycin" artificial intelligence group worked with physicians to develop medical "consultation systems" to be used by less experienced personnel to diagnose and treat infectious diseases (perhaps onboard submarines or in space). We visited a medical clinic once during five years (even though it was in an adjacent building). Years later, I recognized this was a case of "technology push" -- advancing our own interests as researchers -- rather than trying to help other people. If we had been serious about contributing to medicine, we would have helped develop a patient database. The clinic didn't have computers at all, and we thought to bring them "automated physicians."

By the late 1980s the notion of "participatory design" (people you are trying to help join the design team) came into vogue among computer scientists working with social scientists. The ideas first took hold in Great Britain and Europe (particularly Scandinavia thanks to socialist labor movements), and a few places in the US, particularly the University of California at Irvine. A decade later, I finally started to hear computer scientists in the US refer to the "culture" of a work setting and "communities of practice."

At the Institute for Research on Learning, we worked directly with corporate and school sponsors developing prototype tools. Office workers and students used these tools in practical settings, which were studied by anthropologists, documented on video, and analyzed by multidisciplinary teams ("video interaction analysis"). We engaged in such projects for large corporations such as State Farm, NYNEX (in Manhattan, where I first worked with Maarten, Ron, and Mike), Xerox Business Consulting, and Kaiser-Permanente. In these settings, we always had a champion who promoted our work and often co-wrote proposals. We frequently reported directly to vice presidents and R&D directors.

In 1998, I brought the combination of "in situ design," participatory design, participant observation (ethnography), video interaction analysis, and agent-based systems to NASA. I believed that such methods belonged within the Computational Sciences Division (now Intelligent Systems), and resisted the tendency by people to view my work as only "human factors." The confusion is not surprising, given that when I arrived in 1998, NASA like many other technology organizations deliberately separated researchers who develop systems from researchers who study human-systems interaction. At Ames Computational Sciences and Human Factors were two divisions within the Information Sciences Directorate. Each had different funding sources and worked in different arenas (space and aeronautics). (Isn't the label "Division" wonderful?)

By good fortune, Pascal Lee provided opportunities to begin exercising an integrated design and development methodology at Devon Island in the Haughton Mars Project. Rick and I worked together with Charlie Cockell, a geologist, in July 1998, with Rick using a prototype exploration workstation (bungee-corded to an ATV). Rick parked the ATV wherever it could be stably placed, Charlie dug holes nearby, and I studied what happened. Such rudimentary experiments provided inspiration for the Mobile Agents system three years later.

This year, in MDRS 49, we have brought the method of design in the context of use to another level. We developed a prototype system, based on experience using the inverter-generator system at MDRS last year, and are now iterating to improve our rudimentary tools, even as we live within the system we are seeking to improve.

Our development process follows a few simple principles:

1. Start simply -- build a good framework with a few useful functions (e.g., "What is the hab power usage?").
   
   
2. Design for extensibility -- anticipate what aspects of the system will be improved during the experimental on-site period.
   
   
3. Video record use of the system so the ideas can be illustrated to others. (The Crew Activity Analyzer has automated this process on the upper deck.)
   
   
4. Start with access to data, not causal interpretations (e.g., "What is the solar panel output?" vs. "How long will the batteries last?").
   
   
5. Allow inquiries in the language of the non-expert and respond accordingly (e.g., "What is the status of the generator?"; "The generator is not supplying power to the habitat" vs. "The generator input to the inverter is zero amps" or "Channel A equals zero").
   
   
6. Allow experience, not unbridled imagination to dictate new functionalities -- all of the voice commands added to Mobile Agents derive from requirements discovered on-site, not ideas we dreamed up sitting back at Ames. (And it must be said the error conditions we incorporated in the agent system prior to arrival will be far less useful than those we designed last week.)
   
   
7. Use an agent-based architecture to easily incorporate peripheral devices and compose functionality ( e.g., "Take a note for " is a simple addition to the pre-existing voice note function).

Obviously we can't send computer scientists on the next lunar missions. The systems we send to the moon must be proven useful and must work. Much can be learned from Apollo. But the inherent uncertainty of what will be required and what will make operations more efficient shows the special importance of analog facilities and operations like MDRS. Ironically, some managers today are unsure why NASA needs an analog program now, when we don't yet have the systems to test.

But analog facilities and operations are not just about testing, they are especially valuable for discovering requirements, so we know what to build!

The confusion about analog development appears to have several aspects: 1). The misconception that full systems are delivered in a linear way, following a "technology readiness level" path, "transferred" from research centers to operations centers to flights. 2). The related misconception that everything must be tested in a laboratory or that everything can be tested in a laboratory. 3).Not appreciating how exploratory use of prototype tools in real work contexts is a systems development method that combines experimental, scientific work with engineering. 3). Failing to heed the lesson that complex systems need to be brought together into wholes early on ("total systems approach"). Piecemeal design and development often produces "functionally tested" subcomponents that don't constitute a practically useful work system. (Consider that "paper clip" assistant in a well-known word processor.)

Proper investigation of what to build for planetary surface missions requires simulating operations in distributed teams over long periods of time. JPL used this method to refine tools for MER operations in 2003. Gaps in work processes discovered then and during the mission might have be avoided if the operations simulations had started a year or two sooner. To date, funding and near-term focus on getting a spacecraft to its target relegates software and operations design to secondary priority. Consequently, operations costs become unpredictable. Yet, software is the one aspect of a complex system that can be developed and installed during routine operations, after the rovers and astronauts have arrived. So interesting tradeoffs among risks and costs of hardware, software, and operations are possible.

In summary, "design in the context of use" means trying out integrated work systems in practical settings to accomplish work that matters (e.g., maintaining the MDRS power system). This engineering methodology relies on scientific experimentation with controlled protocols and varied conditions, which analog settings can provide.

Indeed, what if we could send the MDRS Crew 49 to the ISS? Wouldn't the Space Station be an ideal place for developing advanced automation and experimenting with different mission support concepts?

Bill Clancey
Commander, MDRS Crew 49