Keywords: design, program planning, context, detail
Breadth In Design Problem Scoping:
Using Insights From Experts to Investigate Student Processes
Because design plays a central role in engineering, it is important for engineering education programs
to prepare students with design skills.
Implications of Findings
By describing both novice and expert approaches to engineering design, researchers are contributing to the
formulation of more specific design learning outcomes that may be addressed in curriculum design and program
planning. The narrative analysis of expert problem scoping behavior suggested a sophisticated approach to
situating problems and solutions in context.
The narrative analysis provided some promising directions for future research. It allowed us to identify
significant themes in their narratives, potentially different from what the novices had tended to highlight
– existing engineered solutions, alternative design solutions, costs and benefits, priorities, and history.
| Even though they were not asked to do so, the experts began to consider
alternative design solutions and wall materials.
|
More interestingly, narrative analysis allowed us to see the relationships between and among experts' ideas
and develop new understandings of what those relationships look like and how they reflect expert designers'
thought processes.
Method and Background
In this study we used two methods for analyzing expert data: verbal protocol analysis (VPA) and narrative analysis.
VPA has been effectively used to describe the design processes employed by engineering students, expert designers,
and expert-novice comparative research. VPA involves asking participants to "think aloud" while performing the
following design task: In the past, the Midwest has experienced massive flooding of the Mississippi River. What
factors would you take into account in designing a retaining wall system for the Mississippi? Each participant
worked individually and had up to 30 minutes to complete the design task. Verbal responses were audio recorded
and transcribed for subsequent coding and interpretation. Narrative analysis entails a close reading of the
transcripts seeking to identify content categories (particularly those not highlighted in the VPA) but also to
trace the logic or structure underlying the respondent's answer. (See full length paper at the link below for a
complete description of methods.)
The expert data came from a subset of four responses purposefully selected from a pre-existing pool of 19
experienced engineers, representing six engineering specializations, identified by their peers at work as expert
designers.
One main objective was to choose the "most expert" participants. This was accomplished by considering years of
professional experience, as well as responses to a "playground design" task, which they completed prior to the
task discussed in this paper. The second objective was to choose participants to maximize diversity of perspectives
and was accomplished by purposefully including one of the women among the four, and selected diverse participants
based on their area of specialization.
The concepts of design detail and design context enable quantification and comparison of the breadth of problem
scoping among expert responses. Ideas focused on the wall or the water from a technical or logistical perspective
were interpreted to be oriented toward the details of the design problem. All other ideas were considered to be
oriented toward the context of the design problem. There was no common pattern among the four experts in terms of
the numbers of segments, or the distribution of these segments across the coding scheme.
The authors then situated this expert data in novice research from the Academic Pathways Study research element of
the NSF-funded Center the Advancement of Engineering Education. A direct comparison of the expert and novice
samples cannot be made because the methods for collecting data resulted in vastly different numbers of segments.
APS data were collected from students at four of the CAEE partner institutions using surveys, structured and
semi-structured interviews, ethnographic observations, and simple engineering tasks. During the first year of
the study, the MWF task was administered to 124 freshmen. Instead of thinking aloud, students were given ten
minutes to write down their answers. The transcripts were segmented and coded the same way as described for the
expert data above.
The data for each expert participant consisted of a transcribed response to the verbal protocol on the Midwest
floods (MWF) design task. Each transcript was segmented into distinct "though units," or ideas. A coding scheme
was applied in order to systematically measure and categorize design processes. Segmented transcripts were coded
on two dimensions of problem scoping breadth: physical location (wall, water, riverbank, or the wider surroundings
beyond) and frame of reference (technical, logistical, natural, or social).
What We Found
None of the experts in the sample used the full 30 minutes they were given to think aloud. The number of coded
segments among the four experts ranges from 20 to 118, with a total of 228 segments. The total number of segments
includes the unique ideas a study participant offered in response to the MWF problem, as well as any "non-applicable"
or “equivalent” segments.
Novices’ ideas were distributed across physical locations in a similar fashion to those of the experts. As with
the experts, about half of the novices’ segments were focused on the wall, twice as many as those focused on the
water. As with the experts, the frame of reference codes are more evenly distributed across the four categories,
but the distribution is somewhat different with novices considering more factors with a natural frame of reference
than experts.
In the narrative analysis of expert data, five kinds of factors stood out, either because they hadn’t seemed as
pronounced in the novice data or because they emerged rather strongly in the expert data:
- The experts suggested looking at existing retaining wall systems and other existing engineered solutions for
flooding.
- Even though they were not asked to do so, the experts began to consider alternative design solutions and wall
materials.
- The experts weighed costs and benefits throughout their narratives.
- The experts considered priorities based on costs and benefits.
- The experts situated their narratives in history. Each narrative had temporality: a past, a present, and in
some cases, a future.
The following observations were also made about the experts’ thinking processes:
- One approach to the MWF was to state a broad factor, and then elaborate and expand upon it by brainstorming
a list of related detailed factors.
- Expert verbal responses often included logical connections among sets of related factors.
- One method of organizing thoughts was to embed a set of related ideas within another set of related ideas.
Further analyzing the transcripts for these four experts and including those of the other experts in the study
should allow us to identify additional ways of thinking about and doing engineering and affirm or elaborate on
the narrative theories presented here. Illuminating the thought processes of expert designers can provide direction
for improving the way we educate novices.
Authors: Andrew Morozov, Deborah Kilgore, and Cynthia Atman
Source: Proceedings of the 2007 American Society for Engineering Education Conference
The full paper, including references, is available via ASEE proceedings search.
For a printable pdf of this research brief, click here.
Brief created July 2007
Back to Research Briefs