Using near-real-time monitoring data from Pu‘u ‘Ō‘ō vent at Kīlauea Volcano for training and educational purposes

Abstract

Training non-scientists in the use of volcano-monitoring data is critical preparation in advance of a volcanic crisis, but it is currently unclear which methods are most effective for improving the content-knowledge of non-scientists to help bridge communications between volcano experts and non-experts. We measured knowledge gains for beginning-(introductory-level students) and novice-level learners (students with a basic understanding of geologic concepts) engaged in the Volcanoes Exploration Program: Pu‘u ‘Ō‘ō (VEPP) “Monday Morning Meeting at the Hawaiian Volcano Observatory” classroom activity that incorporates authentic Global Positioning System (GPS), tilt, seismic, and webcam data from the Pu‘u ‘Ō‘ō eruptive vent on Kīlauea Volcano, Hawai‘i (NAGT website, 2010), as a means of exploring methods for effectively advancing non-expert understanding of volcano monitoring. Learner groups consisted of students in introductory and upper-division college geology courses at two different institutions. Changes in their content knowledge and confidence in the use of data were assessed before and after the activity using multiple-choice and open-ended questions. Learning assessments demonstrated that students who took part in the exercise increased their understanding of volcano-monitoring practices and implications, with beginners reaching a novice stage, and novices reaching an advanced level (akin to students who have completed an upper-division university volcanology class). Additionally, participants gained stronger confidence in their ability to understand the data. These findings indicate that training modules like the VEPP: Monday Morning Meeting classroom activity that are designed to prepare non-experts for responding to volcanic activity and interacting with volcano scientists should introduce real monitoring data prior to proceeding with role-paying scenarios that are commonly used in such courses. The learning gains from the combined approach will help improve effective communications between volcano experts and non-experts during times of crisis, thereby reducing the potential for confusion and misinterpretation of data.

Background

Communication between scientists and non-scientists (e.g. policy makers, the general public) is a significant challenge in volcanology (e.g., McGuire et al. 2009). The two groups may not be familiar with each other’s terminology, even to the extent that some words, such as “tilt” and “eruption”, will have different meanings to different people. Data and scientific reports may be readily available to the general public, but it is unreasonable to expect non-science experts to efficiently navigate such resources, understand data-collection methods and limitations, and comprehend how the data may relate to hazardous events (especially those with low probabilities, like caldera-forming eruptions). Leaving volcano-monitoring data and conclusions open to interpretation by untrained non-scientists (e.g. “non-experts” who are represented here by introductory-level students) has the potential to introduce needless confusion; or in the worst-case scenario, improper action -or lack of action, which during a crisis can have disastrous consequences to life and infrastructure. The consequences of ineffective communication between scientists and policy makers, planners, the media, and the public can be dire (e.g. Fiske 1984; Voight 1990). Effective interaction between scientists and non-scientists regarding data collection and application to eruption forecasting is therefore of critical importance—but how can communication between these groups be bridged?

We addressed this question by testing the utility of online near-real-time volcano-monitoring data as an educational tool in college and university courses, with two populations of students; non-geology majors (referred to as “non-experts”) and geology majors (considered “novices”). Real- and near-real-time datasets, including webcam imagery, earthquake locations, and surface deformation, that are available via the Internet are increasingly common for volcanoes around the world and constitute a valuable, yet largely unrealized, resource for formal and informal geoscience education. Such data demonstrate the dynamic nature of the Earth and are an effective means of connecting with students and other non-experts, especially once they realize that they are looking at the same information- at the same time as professionals who are attempting to better understand volcanic processes. Online datasets (e.g. GEOROC, IODP/Janus, Marine Geoscience Data System, Smithsonian GVN, and other databases), are seldom comprehensive, presented in a format useful for in-class adaptation, or accompanied by background information to aid with interpretation—aspects that are critical for educational applications both in and outside of the classroom.

The Volcanoes Exploration Program: Pu‘u ‘Ō‘ō (VEPP) is a website that was established to address the need for an organized and up-to-date educational resource in volcanology (United States Geological Survey 2009; Poland et al. 2010b,c; Poland et al. 2011). The VEPP website hosts near-real-time monitoring data from the active Pu‘u ‘Ō‘ō eruptive vent on Kīlauea Volcano, Hawai‘i, along with background and context information (Figure 1). A strength of VEPP is the common theme of the Pu‘u ‘Ō‘ō eruption (ongoing since 1983; Heliker and Mattox 2003; Orr et al. in press), which allows the website to be revisited multiple times to demonstrate different principles of eruption monitoring and integration of many types of data used in volcano monitoring.

Figure 1

Example of VEPP website showing option tabs such as Data, Techniques, and Updates.

The VEPP website is home to a variety of data and ancillary materials concerning the Pu‘u ‘Ō‘ō vent of Kīlauea. Near-real-time datasets include webcam imagery, tilt, GPS, and seismic amplitude, while maps of lava flow activity and results from episodic kinematic GPS surveys covering a network of benchmarks around Pu‘u ‘Ō‘ō are made available as those measurements are collected. The time-series data (e.g. seismic amplitude, tilt, and GPS) can be queried through the VALVE (Volcano Analysis and Visualization Environment; Cervelli et al. 2002) interface, which allows a user to plot data from various monitoring stations over user-defined time periods. Webcam imagery is available via a searchable database, while lava flow maps and kinematic GPS results can be downloaded for the specific dates on which they were collected. The data are supported on the VEPP site by information including the history of the Pu‘u ‘Ō‘ō eruption, how various monitoring data are collected, and how monitoring data relate to eruptive activity. The vision for VEPP is that students and other users can examine a diversity of monitoring data from a period that they define—either collected recently or spanning some past event of interest—and use those data to interpret the volcanic processes that are responsible for the observed signals.

We explored the impact of VEPP on improving students’ understanding of volcano monitoring and hazards through the use of a teaching module that simulates the weekly staff meeting at the U.S. Geological Survey’s Hawaiian Volcano Observatory. We developed the “VEPP: Monday Morning Meeting at the Hawaiian Volcano Observatory” (MMM) activity to give students the opportunity to work in small groups to use VEPP data to monitor an eruptive event at Pu‘u ‘Ō‘ō. The goal of this work is to determine if the MMM VEPP activity effectively results in transitioning beginner or novice level students to the advanced level on the spectrum of comprehension for volcano monitoring techniques and if their ability to interact with experts increases.

The learning goals of the activity are for students to be able to “interpret a multidisciplinary dataset for monitoring volcanic activity and to use that data to make a forecast for a potential eruption” (Poland et al. 2010a). We found that exposure to VEPP data through the MMM activity advanced non-expert students to a level where they could communicate effectively with experts. Our results suggest that to improve exchanges between scientists and non-scientists, it is important to gain the perspective of how non-experts think, how they can learn, and how information may best be targeted to different levels of non-experts, which should reduce a source of misunderstanding when scientists and non-scientists interact during a volcanic crisis. The VEPP approach to educating non-experts is unique compared to other training exercises because the student participants work with real volcano-monitoring data, as opposed to mock monitoring results (even though these are usually based on real eruption experiences). Other methods used to enhance communication between groups or the understanding level of non-experts such as tabletop exercises (e.g., Haynes et al. 2008; Solana et al. 2008; Pierson et al. 2013) and direct-delivery courses with role-playing scenarios used in the Federal Emergency Management Agency course on Volcanic Crises Awareness (B. Houghton, written communication, 2014) may benefit from including VEPP or similar online volcano-data resources. Including a VEPP-based activity that utilizes real volcano-monitoring data in such courses prior to an eruption role-playing project would provide important background to beginner and novice participants, allowing them to get more out of the role-playing exercise, and take better advantage of the monitoring experience. The VEPP data and related activities may also work well leading up to other simulation activities such as the eruption crisis simulation of Harpp and Sweeney (2002) or in addition to other volcano-related simulations such as Hales and Cashman (2008) for training or classes focused on diverse volcano hazards.

Methods

VEPP and Monday Morning Meeting exercise

To support use of the VEPP website, a week-long workshop was held at Kīlauea Volcano in July 2010. The 25 participants from the United States and Canada represented a diverse cross section of higher learning institutions, from community colleges to research universities, and they included geology faculty who teach introductory non-major classes (both large and small), laboratories, discipline-specific upper-division courses, and graduate seminars. An outcome of the workshop is that 20 new volcanology teaching modules that incorporate the VEPP site were developed and made available for community use via the Science Education Resource Center (SERC) website (National Association of Geoscience Teachers 2010; Poland et al. 2011). We developed the “VEPP: Monday Morning Meeting at the Hawaiian Volcano Observatory” activity (Poland et al. 2010a) and have assessed how student learning can be improved through the use of online volcano-monitoring data that are supported by background materials and use of effective and research-based pedagogical methods.

The Monday Morning Meeting (MMM) activity is intended to simulate the weekly staff meeting of the USGS Hawaiian Volcano Observatory, during which experts from different disciplines (typically geology, geodesy, seismology, and gas geochemistry) present data collected over the previous week, and discuss implications for volcanic activity. The MMM exercise is a jigsaw activity (Aronson et al. 1978; Shulman and Sherin 2004), in which students in small groups (3–4 students) are given one type of monitoring data, either tilt, GPS, webcams, or seismic data associated with the July 21, 2007, fissure eruption on the east flank of the Pu‘u ‘Ō‘ō eruptive vent (Figure 2; Poland et al. 2008). We chose to use archive data available on the VEPP website because an interpretation of that eruptive activity has already been published, providing the opportunity for classes to complete the MMM exercise to predict volcanic activity and then to read the real outcome (Poland et al. 2008). The same exercise can be run using near-real-time data, in which students interpret volcanic activity as it is happening. Student groups discuss their assigned data type in terms of what the data indicate is happening on the volcano. A handout included in the activity (available at http://nagt.org/nagt/teaching_resources/vepp/examples/48383.html) guides student discussion. After a set period of time for discussion, new groups are formed consisting of at least one student from each discipline. The students describe their data to one another and are then tasked with developing an interpretation of the volcanic activity that can satisfy all the datasets they have examined. Instructors can use a handout included in the activity to guide the second set of student discussions, which can be followed by a whole-group discussion led by the instructor. The timing of each student discussion session can be determined by the instructor, based on our use of the MMM activity, we recommend 30–50 minutes each. As with any jigsaw activity (e.g. Aronson et al. 1978), the distribution of each data type is governed by the number of students and groups participating in the two discussion sessions. Grading is at the discretion of the instructor, an additional writing assignment is provided, which asks student participants to read the EOS-Transactions article that describes the eruption event they had been asked to interpret from the activity (Poland et al. 2008) and compare their final interpretation based on VEPP data with actual events.

Figure 2

Examples of GPS, Tilt, Seismic and Webcam data used in MMM activity for July 1–31, 2007.

Students in seven different undergraduate geoscience courses participated in the MMM activity during 2011–13 (Table 1) and took a pre- and post- MMM activity survey. In each of these classes, learning gains ((Post%-Pre%)/(100-Pre%) from Hake 1998) were measured for students who completed surveys that include multiple choice and open-ended responses, which were qualitatively analyzed for learning (Tables 2 and 3). Participants at the two-year college (2yc) included students in a geologic hazards introductory course, in which students engaged in the activity administered during three separate classes, and in two physical introductory geology courses, in which students engaged in the activity in just one class period. Participants at the comprehensive university (4yc) used the MMM exercise in a large lecture setting for the introductory physical geology course during two class periods and in a small upper-division volcanology course during two class periods. All classes at the 2yc were taught by one instructor and all classes at the 4yc were all taught by a second instructor.

Table 1 Results of MMM multiple-choice question #1

Table 2 Statistical analysis of results of MMM multiple choice question

Table 3 Learning gains for question 1 on MMM survey

Our intent for administering this range of populations, class settings, and time frames was to assess if time, class size, or level of background expertise played a role in the ability for the students who started as non-experts to improve their knowledge of volcano monitoring as a result of using the MMM activity. For the purpose of analyzing learning in the MMM activity, we defined four categories of learners. Introductory students started at a beginner level with minimal amounts of prior knowledge based on pre-MMM survey results. It is this population that we liken to the non-expert, general population (i.e., public decision makers, news media, and the general public) with whom volcanologists may need to interact during times of planning and crisis. The volcanology students had some prior knowledge both from the course content as well as other courses where some of the methodology and content may have been provided in other contexts, such as the use of GPS in a Structural Geology class. We consider these students to be novices; they have more knowledge than a beginner. Advanced learners are able to apply appropriate terminology to the collection and use of specific volcano-monitoring techniques and have a basic ability to interpret the data. As such, the knowledge of advanced learners is similar to what is desired for civil defense personnel and other professionals with whom planning and communication often occurs in volcanically active regions. Experts are volcanologists who actively work with volcano monitoring data. The notion of a beginner- to- expert continuum is similar to those described in other contexts (Chi et al. 1981; Bransford et al. 2000; Petcovic and Libarkin 2007). The goal of this work was to determine if the MMM activity effectively allows for a beginner or novice to advance on the spectrum of comprehension for volcano-monitoring techniques and become more literate in conversing with experts.

Content learning from MMM

Our primary method for measuring learning gains was from the use of a pre-MMM and post-MMM survey administered to students (electronically or paper format, see discussion below) prior to and following the MMM activity (Table 4). The post-MMM survey was administered between 1–3 weeks after the activity. Pre- MMM- and post- MMM surveys were delivered in hard copy during all three classes at the 2yc (GLG111, GLG101a, and GLG101b) and the 2011 volcanology course at the 4yc (GEOS436). Electronic versions were offered to students in two introductory courses (GEOS101a and GEOS101b) and the 2013 volcanology course at the 4yc. Not all enrolled students completed both the pre-MMM and post- MMM surveys offered electronically, with response rates of 28% (spring 2013) and 47% (fall 2012) for the introductory courses and 70% for the volcanology students. While the response rate in the spring 2013 introductory class was low, respondents received the same range of final grades as the whole class, and their survey scores were statistically consistent with the fall 2012 class (which had a 47% response rate).

Table 4 Questions from the pre-MMM and post-MMM surveys

The MMM curriculum and survey were developed through an iterative process of consultation between content experts and education specialists. The MMM survey is tied to the content taught within the MMM module and is designed to capture student learning gains about methods used to monitor and forecast volcanic eruptions through multiple questions.

The multiple-choice question in both the pre- MMM and post- MMM survey asked students to identify which of “the following data types are used to monitor volcanic eruptions?” Possible answers included widely used methods and distractor responses that are not commonly used for monitoring purposes. The scoring for this question was one point for each correctly identified method and a loss of a point for each inaccurate method identified and each unidentified correct method. The lowest possible score was −8 and the highest possible score was +4. We completed basic statistical tests (number of participants, pre-MMM and post- MMM survey scores, mean, and standard deviation) and t-test comparisons for each class and between beginner (introductory classes) and novice (volcanology classes) levels to assess prior knowledge and overall learning gains for all student participants. Identification of beginner, novice, and advanced level learners are for the purposes of distinguishing learner types before and after the use of the MMM activity and do not imply knowledge beyond the scope of the pre-MMM and post- MMM surveys completed here.

In addition to the multiple-choice question, we measured student responses to open-ended questions about the collection and use of GPS, tilt, and seismic data, which provided greater insights into student understanding of these concepts (Table 5). Student participants were asked to briefly describe how each of the three monitoring data types (GPS, tilt, seismic) was collected and used to monitor volcanoes. A summative, short-answer question asked students to, “Describe and interpret changes in tilt patterns expected before, during, and after an episode of lava effusion at Pu‘u ‘Ō‘ō (a volcanic vent of Kīlauea)”. This last question provided students the opportunity to tell a more holistic story of events associated with volcanic activity in the context of a specific method used for volcano monitoring and forecasting.

Table 5 Results of MMM Written results from pre-MMM and post- MMM surveys

Open-ended responses from students were analyzed for overall learning gains by comparing responses from pre-MMM to post- MMM surveys. Changes in learning scores were determined from a thematic level of analysis (Miles and Huberman 1994). Thematic analysis is an approach of looking at trends within the data to identify responses used by students that illustrate common learning themes. Common phrases revealed two themes across the student responses: a) use of vocabulary, through increased use of appropriately applied terminology (e.g., inflation/deflation, radial/tangential tilt) or b) overall conceptual gains when pre-MMM and post-MMM responses from the same student were compared. Answers that indicated an increase in understanding (e.g., a pre-MMM survey answer of “I don’t know” or “seismographs” in response to the question of how seismic data are collected would be considered improved if the student’s post-survey response was more detailed, such as, “using seismometers to detect an increase in earthquake activity” [student A34]) were scored as +1. Responses that were categorized as the same or no change in understanding received a score of “0”. Responses received a −1 score if there was evidence of a decrease in understanding or use of language, or if they reflected a misconception that was not evident in the pre-MMM survey. For example, one student (10–394) applied the term “potentiometric GPS” in the post-survey response, which was one of the distractors in the pre-MMM and post- MMM survey multiple-choice question. Responses to the open-ended questions were initially scored separately by the individual instructors and were subsequently evaluated jointly. If there was a disagreement in the scoring, a common score was agreed upon through discussion. It was possible for a student to improve their understanding and also gain a new misconception, so there were some students who received both a +1 and −1 score. Counts of values were used to determine percent of learning gains, loss, or no changes and correlated to student learning gains for the multiple choice question results. The total possible range of change from pre-MMM to post- MMM survey responses to open-ended questions is +7 (increase in all seven questions) to −7 (decrease in all seven questions).

Confidence development from MMM

Self-efficacy is the confidence in one’s ability to be successful with a specific task in the ability to understand content largely based on prior experiences (Bandura 1986; Zimmerman 2000), and it consistently predicts student performance in different subjects, including introductory science courses (e.g., Chemistry, Zusho et al. 2003; Geology, McConnell et al. 2010). While self-report data is dependent on the experiences and prior knowledge of the student, such data are valuable in helping to determine changes in the perception of student learning, and can be predictive of performance (Assor and Connell 1992). Self-efficacy of student participants was measured in pre-MMM and post- MMM surveys from self-ratings on a Likert Scale (Likert 1932) for statements, “I understand [seismic/tilt/GPS/webcam] data and how to interpret it regarding eruptive conditions”. An additional set of data are from self-ratings on a Likert Scale for the statement, “I understand that assessments of volcanic activity are based on multiple datasets, and can describe how different datasets complement one another”. In all cases, students were asked to rate their level of confidence using a scale of 1 (not at all), 3 (somewhat), 5 (I could look at this data and assess the state of activity). Changes in self-efficacy are from comparison of individuals’ scores in their pre-MMM and post- MMM survey responses (Table 6).

Table 6 Results of MMM self-efficacy scores for pre-MMM and post- MMM surveys

Results and discussion

Content learning: Multiple choice

In order to compare the introductory students (beginners) to students enrolled in the volcanology course (novices), we compared the mean pre MMM values for both populations. The pre MMM survey for the beginners was −1.56 (normalized to 6.44) and for the novices was 2.00 (normalized to 10.00). An independent t-test comparison between these two populations determined that the pre-MMM survey responses for the beginners was significantly different from the novice pre-MMM survey results (t = −11.16, df = 36.28, p < 0.001, with non-equal variances assumed due to the large difference in population sizes). The effect size illustrates the magnitude of the difference between these populations with a 1.89 Cohen’s d-value, where general convention is that anything over 0.8 is considered to be a large effect size (Cohen 1969). As a result, we have including a general category of “introductory” that treats all of the introductory students at both the two-year college and the four-year college as one population.

An independent t-test compares the means between the two populations in order to determine if the difference in means is significantly different than zero (Coladarci et al. 2008). A paired t-test compares the difference in mean scores for each participant and determines if that difference is significantly different from zero (Coladarci et al. 2008). Cohen’s d is a measure of the effect size, which is to say, how much the means between two populations vary as a function of their standard deviations, this measure is more meaningful than significance which is a function of the size of a given population (Coe 2002). Learning gains are calculated by normalizing the pre and post scores out of 12, so a pre score of −3 would result in a value of a normalized score of 5 and would be the equivalent of a pre % score of 38.46%. As a result, learning gains were calculated as a function of the formula from Hake (1988) of (Post%-Pre%)/(100-Pre%). This allows one to examine the learning gains as a function of the maximum possible learning gains. As such, it has previously been posited that any gain over 0.7 is a large learning gain, between 0.7 and 0.3 is a medium learning gain and less than 0.3 is a small learning gain (Hake 1998). Table 3 illustrates the learning gain means for each class. Figure 3 illustrates the learning gains, converted to percent as compared to the initial pre-score values, which illustrates the high incoming knowledge of the volcanology (novice) students. The results of this difference in incoming population supports the importance of the follow-up details from the written responses in an effort to better understand what these learning gains represent in a more nuanced fashion.

Figure 3

% Learning gains vs. % Pre-MMM score gains, which illustrates the overall learning gains for most student participants, even with the higher pre-scores for the volcanology (novice) students.

In identifying monitoring methods in the multiple-choice question, all of the beginners (introductory) had significant changes in their MMM survey scores. While the extent of learning for novices (volcanology) was not statistically significant, both the small sample size and high pre-MMM survey score may have impacted this result. Beginner learners increased their scores from −1.56 to 2.21 and novice learners increased scores from 2.00 to 2.77 (Table 1). Novice-student scores indicate that they started where the beginner’s knowledge ended. Because this item only scored to a maximum of 4 points, the total growth for the novice learners was limited. Students who started at the novice level may have experienced a ceiling effect on the multiple choice question of the post-MMM survey in that their pre-MMM survey scores were high, which may have limited how high their post MMM score could improve (e.g., Deslauriers et al. 2011). This limitation in demonstrating learning was reinforced with the learning gains, which were both positive, but was 0.53 for the beginner population and 0.14 for the novice population on a scale from −1 to +1 (Table 3).

Content learning: open-ended responses

For the purposes of this paper, in which we are examining the learning of two populations —beginners and novices—we find that the open-ended responses reveal more than the quantitative results from the multiple choice question analyzed. In particular, we note different degrees of shifts in understanding of the content between the pre-MMM and post- MMM survey. Students shifted from a beginner to a novice level of understanding and from a novice to an advanced level of understanding. Examples of these shifts are presented below, selected to illustrate shifts in learning across a range of monitoring topics. For all open-ended response questions, minor spelling errors were corrected provided they did not change the meaning of the student’s statement (e.g., “erution” was corrected to “eruption” [student 10–404], whereas “satiles” was not changed to “satellites” [A-7] as it may or may not represent an understanding of satellites).

Beginner learning gains (introductory students)

Introductory students, who generally lacked prior knowledge of volcano monitoring, had small improvements in understanding, improving, on average, by 1 point. We categorize this shift as beginner to novice. While their understanding grew, it lacked the larger conceptual framework within which volcano-monitoring data are used and inform interpretations; however, these shifts are important for students who are starting from the lowest level of experience and understanding. Examples of student responses are presented below to illustrate that students often started with little to no knowledge about the subject, and in some cases started with incorrect knowledge, but gained a greater familiarity with the data-collection method and how it is used to monitor volcanoes after participating in the MMM activity. Pre- MMM and post-MMM survey responses below are from students in introductory courses at the two year college (class: GLG101) for questions that ask how data are collected (question a) and how they are used in volcano monitoring (question b).

GPS Data Example Response:

Pre: “a) satellites, b)?”

Post: “a) Through certain points that change GPS location during an eruption, b) A shift in the position” [A39]

Tilt Data Example Response:

Pre: “a) unsure, b) unsure”

Post: “a) The data is collected through a machine that can detect if the magma underground is moving, b) The sign that a volcano might erupt would be if the tilt was increasing and increasing all around the volcano”. [A37]

Seismic Data Example Response:

Pre: “a) minerals, b) I don’t know”.

Post: “a) by how much the earth shake, b) the graphs it comes out with” [A45]

These kinds of small shifts in understanding, while not completely correct, represent partial shifts similar to those seen in other beginner populations engaged in active-learning scenarios (Lewis et al. 2010).

In some cases, beginner students who had some prior knowledge demonstrated a more obvious shift in learning. An example of this prior knowledge from a 2yc introductory-course student is:

Seismic Data Example Response:

Pre: “a) seismic waves are collected from plate movement; b) a sign would be significant seismic waves (an earthquake)”

Post: “a) seismic data is collected by use of seismographs, place strategically around or on the volcano; b) A change in earthquake activity tends to signal an impending eruption, so if a change or increase in earthquake activity would occur, it would signal an impending eruption”. [A26]

This example illustrates how possessing prior knowledge on a topic can lead to greater advances in overall conceptual understanding (Chinn and Brewer 1993; Lewis et al. 2010).

Students in the 2yc introductory hazards course (GLG111) had more time (200 minutes vs 100 or 75 minutes spent in other courses) to spend with the MMM activity, which led to opportunities to engage more deeply with the content and allowed for greater overall shifts in learning gains. Examples below represent responses for students from the introductory hazards course:

GPS data example response

Tilt Data Example Response:

Pre: “a) GPS trackers are placed on the volcano to see if it is moving or shifting, b) unsure”,

Post: “a) GPS nodes are placed around the volcano and send information of their location to a satellite and then to us, b) If there is increased or decreased distance between the GPS nodes then you know it is active/possible eruption”. [A8]

Pre: “a) I don’t know, b) I don’t know”,

Post: “a) Using tiltmeters on or just below the surface, they measure if the ground is ‘tilting’ up or out (radial-up, tangential-out), b) The tiltmeters will show that the magma underneath is expanding, causing the ground to balloon out”. [A5]

This student [A5] also greatly improved the explanation of how tilt data varies before, during, and after an eruption with:

Pre: “I don’t know”

Post: “Before: The tilt patterns showed radial and tangential tilt prior to the eruption, showing that the magma was expanding, During: The tilt patterns showed more radial tilt than tangential, due to the pressure of the magma being released, After: The tilt patterns lowered and show little sign of tilt”. [A-5]

Responses like these suggest that the amount of time spent on the activity may help support beginners in developing more sophisticated understanding. Both prior knowledge and increased time spent on the MMM activity are important factors that should be considered in the development of training courses or other situations that involve working with non-experts.

Novice learning gains (volcanology students)

Novice learners (volcanology students in GEOS436) start the MMM activity with greater prior knowledge (pre-MMM survey multiple choice average scores = 2, Table 1) and fewer “I don’t know” responses than beginner learners (e.g., introductory students), although the learning gains of novices are lower for the multiple choice question (0.14) than for beginners (0.53). Novice learners outperform beginner learners in conceptual knowledge following the MMM activity, as measured by the average increase in open-ended (written) responses to questions about the collection and use of monitoring data—novice learners improved on average by 3.4 points (compared to 1 point for beginner learners). These results indicate that the novice learners increased their knowledge to what we have defined as “advanced” levels (able to apply appropriate terminology to the collection and use of specific volcano-monitoring techniques and have a basic ability to interpret the data) as a result of the MMM activity. For example, student 10-404’s pre-MMM survey answer about the use of GPS data, “GPS data points are used as a source to determine the amount of tectonic movement within a specific area” reflects knowledge of GPS in the context of monitoring tectonic displacements, but is not phrased in the context of volcano monitoring. Following the MMM activity, this student’s response to the same question in the post-survey is, “The increased movement between two specific data points are imminent signs of an impending eruption.” In this case, the student began the activity with a grasp of how GPS is used, but increased the sophistication of the post-survey answer by providing additional detail with regard to the volcanological applications of GPS.

The open-ended responses indicate that the novice learners’ knowledge level becomes much more sophisticated and nuanced and represents a shift towards that of advanced learners. This is an important consideration in the design of training for civil-defense or emergency-planning personnel who might already be familiar with using scientific data but not in the context of a volcanic crisis.

Measures of confidence: beginner

Based on self-efficacy responses in the pre-MMM and post- MMM surveys (Table 6), student confidence in the use of each type of volcano-monitoring data increased for all beginner learners (introductory classes) following the MMM activity. Average increases for beginners range from 1.5 (seismic data) to 2.1 (webcam data). The webcam change may be an indicator of the simplicity in applying webcams to volcano monitoring and research. While most students are likely familiar with webcams, they initially lacked the confidence in applying them to volcano-monitoring scenarios, but were able to rapidly grasp the applications once they were exposed to the data. Efficacy increases were well aligned with increased knowledge in this beginner population. For example, pre-MMM survey responses to questions of how monitoring data are collected and used were left blank or had “I don’t know” responses. In particular, 55–100% of students in the introductory hazards class (GLG111) started with “I don’t know” responses (depending on the particular monitoring method), but in the post-survey, only 40% or fewer responded with “I don’t know” for the same questions.

Measures of confidence: novice

The confidence of novice learners (volcanology students) starts higher than for beginner learners prior to the MMM activity, which is consistent with novices starting from more sophisticated content knowledge. Confidence ratings for most novice learners increased following the MMM activity, with gains ranging from average 0.35 points (on the 5 point scale) for webcam data to average gains of 1.2 for tilt data (Table 6). Comparisons of pre-MMM and post- MMM self-efficacy data reveal lower increases for novices than for beginner learners for every data type. This may be the result of a ceiling effect in which novices are more confident with data prior to the MMM activity, resulting in smaller changes following the activity—in other words, they do not have as far to go in their confidence rankings in the post-survey. When compared with their learning gains in the written pre-MMM and post- MMM scores, novice learners show much larger improvement than beginner learners, so we suspect that with the MMM activity, novice learners increase their knowledge to the advanced level and come to recognize there is more to know about a particular monitoring technique than they first realized. Their post-survey responses, therefore, reflect a more-sophisticated understanding that the monitoring techniques are complex. In particular, as they gain a greater understanding of where their ability is on the expert continuum, they may be less likely to indicate confidence at a full level (score of 5), which may also dampen the impact of self-efficacy scores with the novice population. As learners recognize that the use of monitoring data is more complex and contains more uncertainty than they previously thought, they may be more cautious as they assess their ability to use the data for volcano-monitoring purposes—in essence, as students have better models to construct their understanding, they can more accurately gauge their abilities and limitations (Bandura 1986).

Connections between learning and self-efficacy with MMM

Post-MMM survey scores for open-ended questions show improved content knowledge accompanied by improved self-efficacy measured for all classes (Figure 4). Beginner learners all start the MMM activity with similarly low self-efficacy in seismic, tilt, and GPS data, but high confidence in the use of webcam data. Post-data indicate that all classes increased in self-efficacy for all types of data, with the largest increase for webcam data. The multiple-choice scores for beginner learners increased by 22-29%, which is nearly the same as increases in their Likert-scale scores, by 22%-30% from before to after using the MMM activity (Tables 1 and 6).

Figure 4

Comparison of multiple choice (MC) and Self-Efficacy scores in post-surveys, after students use MMM activity. Classes are designated by colors as indicated, monitoring data types are designated by symbols.

A prevailing informal assumption is that training/teaching in smaller class sizes results in greater learning for beginners, but there is evidence that with the MMM activity, learning can happen just as well with beginners in larger classes (e.g. >100 students) as in smaller classes (e.g., 13 students), particularly if enough time is spent on the activity. Figure 5 illustrates that multiple-choice learning gains occurred in all ranges of class sizes (from 13 to 105 students).

Figure 5

Increase in learning from multiple choice and written responses for classes based on number of students enrolled.

Class time spent on the activity may be an important factor for increasing student learning gains (Figure 6), which is particularly apparent with the open-ended questions. Students who had at least 100 minutes to spend on the activity at the beginner level were more likely to have higher overall learning gains than beginners who spent 75 minutes. This becomes an important consideration when introducing non-experts to volcano-monitoring methods. Brief overviews and one-time exposures are probably insufficient if attempting to shift beginners’ understanding to a novice level. Longer time periods spent with the MMM module were also correlated with stronger knowledge retention. Beginner learners who spent at least 100 minutes on the MMM activity retained their ability to identify monitoring techniques at a 98% success rate for the final exam, which was three weeks after the conclusion of the activity. Of that group, 88% were able to identify how inflation and deflation patterns related to eruptive activity on the final exam. This suggests that the MMM scenario provides a meaningful experience for beginners to learn about volcano monitoring beyond a superficial awareness. For learners that started at the novice level, the shift to an advanced level following the MMM activity argues that increased background knowledge (Chinn and Brewer 1993; Zimmerman 2000) and increased exposure to information enhances learning and efficacy levels (Pintrich and Zusho 2007).

Figure 6

Increase in learning from multiple choice and written responses for classes based on amount of time (in minutes) spent on the MMM activity.

Conclusions and recommendations

Themes of MMM training

Beginners (introductory students) show the greatest learning gains in the use of volcano-monitoring data but showed smaller gains in open-ended written responses, which represents their lower-level ability to articulate new-found knowledge. Scores on Pre-MMM surveys reflect their unfamiliarity and lack of confidence in the use of scientific volcano monitoring data, but their increase to the novice level with very large learning gains (0.38-0.59) demonstrates their improved understanding of information provided by expert scientists.

Our results reinforce the idea that novice learners (volcanology students) with some prior knowledge improve to advanced levels of understanding through use of the MMM activity. Novice learners had a stronger shift to advanced conceptual understanding (especially in comparison to beginners) and also appear to realize the limits of their knowledge, which may explain the smaller increases in self-efficacy scores from those of beginners.

An overriding theme of our work with the MMM activity in college-level courses is that the more time that is spent with the module, the greater the learning gains and retention, especially for beginner learners. This result supports the idea that training intended to prepare non-experts to deal with volcanic activity should dedicate an appropriate amount of time to the subject. Based on the learning and the increase in self-efficacy documented here, using an activity like the VEPP MMM in situations where participants have the opportunity to work with real monitoring data from an active volcano is a good complement to eruption scenario exercises. For beginner students, using archived data that are associated with a well-characterized past event may be preferable so they can compare their interpretations to the actual volcanic activity (in other words, an “answer” already exists) in Poland et al. 2008. More advanced students would benefit from using near-real-time data (also available on the VEPP website), allowing their interpretations to evolve over time along with the activity they are tracking, and the uncertain nature of the outcome would be a lesson in itself. This use of near-real time-data and the sophisticated nuances that come with it could address some of the ceiling effect observed in the novice population.

Recommendations

Based on our experience using VEPP and the MMM activity with beginning- and novice-level undergraduate students, we recommend that those who engage in volcano-monitoring training programs for non-science experts such as emergency managers, land use planners, members of the media and civil defense experts consider the following:

• 1. Volcano-monitoring reports and data are currently made available to the public but aren’t necessarily incorporated into training modules designed to familiarize beginners with the use or understanding of those data (e.g Kokelaar 2002, Setting, Chronology and consequences of the eruption of Soufrière Hills Volcano, Montserrat 1995–1999 Activities like MMM provide beginners with instruction and active engagement that guide them through the interpretation of real data, resulting in improved learning. The fact that working with real volcano-monitoring data is clearly tied to increases in understanding of how such data can be used to assess volcanic unrest argues that the practice should be integrated into training that targets improving communication between scientists and non-scientists. For example, using the MMM or similar activity based on real monitoring data could be used in exercises that bring together volcanologists and land managers, emergency response officials, members of the media (who relate reports of volcanic activity to the general public), and others whose work may involve understanding hazards and risk (e.g. insurance agents). When used in conjunction with role-playing scenarios especially, non-experts are likely to increase their comprehension and retention of volcano monitoring with the MMM activity. We therefore recommend that activities similar to MMM be developed for other volcano-eruption scenarios, and that such an approach may be useful in other hazard situations (e.g., responses to earthquakes, tsunami, and severe weather).

• 2. The amount of time that participants (and beginners in particular) are engaged in training activities is critical to their success. Beginners who engaged with the MMM activity for 100 minutes or more showed improved learning over those who had less time with the activity. We therefore recommend that training activities like MMM should be of sufficient time for trainees to engage with data in a meaningful way.

• 3. Experts (volcanologists, in this case) should be cognizant of the level of understanding of their audience when communicating volcano-monitoring information to avoid confusing or unintentionally overwhelming non-expert media, civil-defense, or planning personnel. Based on learning observed from MMM, we recommend that beginner- and novice-level populations of civil-defense and other public safety officials or media personnel be trained in the interpretation of volcano-monitoring data through activities that use real data in an interactive format such as MMM and others (e.g. Hales and Cashman 2008, the FEMA Volcanic Crises Awareness).