Using near-real-time monitoring data from Pu‘u ‘Ō‘ō vent at Kīlauea Volcano for training and educational purposes

Content learning: Multiple choice

In order to compare the introductory students (beginners) to students enrolled in the volcanology course (novices), we compared the mean pre MMM values for both populations. The pre MMM survey for the beginners was −1.56 (normalized to 6.44) and for the novices was 2.00 (normalized to 10.00). An independent t-test comparison between these two populations determined that the pre-MMM survey responses for the beginners was significantly different from the novice pre-MMM survey results (t = −11.16, df = 36.28, p < 0.001, with non-equal variances assumed due to the large difference in population sizes). The effect size illustrates the magnitude of the difference between these populations with a 1.89 Cohen’s d-value, where general convention is that anything over 0.8 is considered to be a large effect size (Cohen 1969). As a result, we have including a general category of “introductory” that treats all of the introductory students at both the two-year college and the four-year college as one population.

An independent t-test compares the means between the two populations in order to determine if the difference in means is significantly different than zero (Coladarci et al. 2008). A paired t-test compares the difference in mean scores for each participant and determines if that difference is significantly different from zero (Coladarci et al. 2008). Cohen’s d is a measure of the effect size, which is to say, how much the means between two populations vary as a function of their standard deviations, this measure is more meaningful than significance which is a function of the size of a given population (Coe 2002). Learning gains are calculated by normalizing the pre and post scores out of 12, so a pre score of −3 would result in a value of a normalized score of 5 and would be the equivalent of a pre % score of 38.46%. As a result, learning gains were calculated as a function of the formula from Hake (1988) of (Post%-Pre%)/(100-Pre%). This allows one to examine the learning gains as a function of the maximum possible learning gains. As such, it has previously been posited that any gain over 0.7 is a large learning gain, between 0.7 and 0.3 is a medium learning gain and less than 0.3 is a small learning gain (Hake 1998). Table 3 illustrates the learning gain means for each class. Figure 3 illustrates the learning gains, converted to percent as compared to the initial pre-score values, which illustrates the high incoming knowledge of the volcanology (novice) students. The results of this difference in incoming population supports the importance of the follow-up details from the written responses in an effort to better understand what these learning gains represent in a more nuanced fashion.

In identifying monitoring methods in the multiple-choice question, all of the beginners (introductory) had significant changes in their MMM survey scores. While the extent of learning for novices (volcanology) was not statistically significant, both the small sample size and high pre-MMM survey score may have impacted this result. Beginner learners increased their scores from −1.56 to 2.21 and novice learners increased scores from 2.00 to 2.77 (Table 1). Novice-student scores indicate that they started where the beginner’s knowledge ended. Because this item only scored to a maximum of 4 points, the total growth for the novice learners was limited. Students who started at the novice level may have experienced a ceiling effect on the multiple choice question of the post-MMM survey in that their pre-MMM survey scores were high, which may have limited how high their post MMM score could improve (e.g., Deslauriers et al. 2011). This limitation in demonstrating learning was reinforced with the learning gains, which were both positive, but was 0.53 for the beginner population and 0.14 for the novice population on a scale from −1 to +1 (Table 3).

Content learning: open-ended responses

For the purposes of this paper, in which we are examining the learning of two populations —beginners and novices—we find that the open-ended responses reveal more than the quantitative results from the multiple choice question analyzed. In particular, we note different degrees of shifts in understanding of the content between the pre-MMM and post- MMM survey. Students shifted from a beginner to a novice level of understanding and from a novice to an advanced level of understanding. Examples of these shifts are presented below, selected to illustrate shifts in learning across a range of monitoring topics. For all open-ended response questions, minor spelling errors were corrected provided they did not change the meaning of the student’s statement (e.g., “erution” was corrected to “eruption” [student 10–404], whereas “satiles” was not changed to “satellites” [A-7] as it may or may not represent an understanding of satellites).

Beginner learning gains (introductory students)

Introductory students, who generally lacked prior knowledge of volcano monitoring, had small improvements in understanding, improving, on average, by 1 point. We categorize this shift as beginner to novice. While their understanding grew, it lacked the larger conceptual framework within which volcano-monitoring data are used and inform interpretations; however, these shifts are important for students who are starting from the lowest level of experience and understanding. Examples of student responses are presented below to illustrate that students often started with little to no knowledge about the subject, and in some cases started with incorrect knowledge, but gained a greater familiarity with the data-collection method and how it is used to monitor volcanoes after participating in the MMM activity. Pre- MMM and post-MMM survey responses below are from students in introductory courses at the two year college (class: GLG101) for questions that ask how data are collected (question a) and how they are used in volcano monitoring (question b).

These kinds of small shifts in understanding, while not completely correct, represent partial shifts similar to those seen in other beginner populations engaged in active-learning scenarios (Lewis et al. 2010).

In some cases, beginner students who had some prior knowledge demonstrated a more obvious shift in learning. An example of this prior knowledge from a 2yc introductory-course student is:

This example illustrates how possessing prior knowledge on a topic can lead to greater advances in overall conceptual understanding (Chinn and Brewer 1993; Lewis et al. 2010).

Students in the 2yc introductory hazards course (GLG111) had more time (200 minutes vs 100 or 75 minutes spent in other courses) to spend with the MMM activity, which led to opportunities to engage more deeply with the content and allowed for greater overall shifts in learning gains. Examples below represent responses for students from the introductory hazards course:

GPS data example response

Responses like these suggest that the amount of time spent on the activity may help support beginners in developing more sophisticated understanding. Both prior knowledge and increased time spent on the MMM activity are important factors that should be considered in the development of training courses or other situations that involve working with non-experts.

Novice learning gains (volcanology students)

Novice learners (volcanology students in GEOS436) start the MMM activity with greater prior knowledge (pre-MMM survey multiple choice average scores = 2, Table 1) and fewer “I don’t know” responses than beginner learners (e.g., introductory students), although the learning gains of novices are lower for the multiple choice question (0.14) than for beginners (0.53). Novice learners outperform beginner learners in conceptual knowledge following the MMM activity, as measured by the average increase in open-ended (written) responses to questions about the collection and use of monitoring data—novice learners improved on average by 3.4 points (compared to 1 point for beginner learners). These results indicate that the novice learners increased their knowledge to what we have defined as “advanced” levels (able to apply appropriate terminology to the collection and use of specific volcano-monitoring techniques and have a basic ability to interpret the data) as a result of the MMM activity. For example, student 10-404’s pre-MMM survey answer about the use of GPS data, “GPS data points are used as a source to determine the amount of tectonic movement within a specific area” reflects knowledge of GPS in the context of monitoring tectonic displacements, but is not phrased in the context of volcano monitoring. Following the MMM activity, this student’s response to the same question in the post-survey is, “The increased movement between two specific data points are imminent signs of an impending eruption.” In this case, the student began the activity with a grasp of how GPS is used, but increased the sophistication of the post-survey answer by providing additional detail with regard to the volcanological applications of GPS.

The open-ended responses indicate that the novice learners’ knowledge level becomes much more sophisticated and nuanced and represents a shift towards that of advanced learners. This is an important consideration in the design of training for civil-defense or emergency-planning personnel who might already be familiar with using scientific data but not in the context of a volcanic crisis.

Measures of confidence: beginner

Based on self-efficacy responses in the pre-MMM and post- MMM surveys (Table 6), student confidence in the use of each type of volcano-monitoring data increased for all beginner learners (introductory classes) following the MMM activity. Average increases for beginners range from 1.5 (seismic data) to 2.1 (webcam data). The webcam change may be an indicator of the simplicity in applying webcams to volcano monitoring and research. While most students are likely familiar with webcams, they initially lacked the confidence in applying them to volcano-monitoring scenarios, but were able to rapidly grasp the applications once they were exposed to the data. Efficacy increases were well aligned with increased knowledge in this beginner population. For example, pre-MMM survey responses to questions of how monitoring data are collected and used were left blank or had “I don’t know” responses. In particular, 55–100% of students in the introductory hazards class (GLG111) started with “I don’t know” responses (depending on the particular monitoring method), but in the post-survey, only 40% or fewer responded with “I don’t know” for the same questions.

Measures of confidence: novice

The confidence of novice learners (volcanology students) starts higher than for beginner learners prior to the MMM activity, which is consistent with novices starting from more sophisticated content knowledge. Confidence ratings for most novice learners increased following the MMM activity, with gains ranging from average 0.35 points (on the 5 point scale) for webcam data to average gains of 1.2 for tilt data (Table 6). Comparisons of pre-MMM and post- MMM self-efficacy data reveal lower increases for novices than for beginner learners for every data type. This may be the result of a ceiling effect in which novices are more confident with data prior to the MMM activity, resulting in smaller changes following the activity—in other words, they do not have as far to go in their confidence rankings in the post-survey. When compared with their learning gains in the written pre-MMM and post- MMM scores, novice learners show much larger improvement than beginner learners, so we suspect that with the MMM activity, novice learners increase their knowledge to the advanced level and come to recognize there is more to know about a particular monitoring technique than they first realized. Their post-survey responses, therefore, reflect a more-sophisticated understanding that the monitoring techniques are complex. In particular, as they gain a greater understanding of where their ability is on the expert continuum, they may be less likely to indicate confidence at a full level (score of 5), which may also dampen the impact of self-efficacy scores with the novice population. As learners recognize that the use of monitoring data is more complex and contains more uncertainty than they previously thought, they may be more cautious as they assess their ability to use the data for volcano-monitoring purposes—in essence, as students have better models to construct their understanding, they can more accurately gauge their abilities and limitations (Bandura 1986).

Connections between learning and self-efficacy with MMM

Post-MMM survey scores for open-ended questions show improved content knowledge accompanied by improved self-efficacy measured for all classes (Figure 4). Beginner learners all start the MMM activity with similarly low self-efficacy in seismic, tilt, and GPS data, but high confidence in the use of webcam data. Post-data indicate that all classes increased in self-efficacy for all types of data, with the largest increase for webcam data. The multiple-choice scores for beginner learners increased by 22-29%, which is nearly the same as increases in their Likert-scale scores, by 22%-30% from before to after using the MMM activity (Tables 1 and 6).

A prevailing informal assumption is that training/teaching in smaller class sizes results in greater learning for beginners, but there is evidence that with the MMM activity, learning can happen just as well with beginners in larger classes (e.g. >100 students) as in smaller classes (e.g., 13 students), particularly if enough time is spent on the activity. Figure 5 illustrates that multiple-choice learning gains occurred in all ranges of class sizes (from 13 to 105 students).

Class time spent on the activity may be an important factor for increasing student learning gains (Figure 6), which is particularly apparent with the open-ended questions. Students who had at least 100 minutes to spend on the activity at the beginner level were more likely to have higher overall learning gains than beginners who spent 75 minutes. This becomes an important consideration when introducing non-experts to volcano-monitoring methods. Brief overviews and one-time exposures are probably insufficient if attempting to shift beginners’ understanding to a novice level. Longer time periods spent with the MMM module were also correlated with stronger knowledge retention. Beginner learners who spent at least 100 minutes on the MMM activity retained their ability to identify monitoring techniques at a 98% success rate for the final exam, which was three weeks after the conclusion of the activity. Of that group, 88% were able to identify how inflation and deflation patterns related to eruptive activity on the final exam. This suggests that the MMM scenario provides a meaningful experience for beginners to learn about volcano monitoring beyond a superficial awareness. For learners that started at the novice level, the shift to an advanced level following the MMM activity argues that increased background knowledge (Chinn and Brewer 1993; Zimmerman 2000) and increased exposure to information enhances learning and efficacy levels (Pintrich and Zusho 2007).