Educational Research and Evaluation
The College of Exploration has provided evaluation leadership and support to a variety of STEM education projects at the national level over the past 15 years. Our experience and expertise include (but are not limited to):
- Longitudinal Research
- Needs Assessment
- Program Evaluation
- Project Evaluation Guidance
- Advisory and Technical Support
- Survey Design
- Lesson Plan and Educational Module Review
- Focus Group implementation
- Developmental Evaluation
Longitudinal Research: The National Ocean Sciences Bowl http://www.nosb.org/
One long-term project is a longitudinal research study for the Consortium for Ocean Leadership’s National Ocean Sciences Bowl since 1999. This study has looked holistically at the entire NOSB program and its effects on a variety of key audiences, but in particular it follows student alumni who participate in the competition. One key goal has been to track these alumni participants as they move into college and then on to careers. The College of Exploration has managed to keep in touch with more than 500 alumni who regularly provide our team with information about their college choices, coursework, majors and career choices. For this NOSB research study TCOE has developed dozens of survey instruments and multiple interview protocols to guide project development. Multiple juried publications have emerged from this body of work including:
Bishop, Tina & Walters, Howard D. (2007). The National Ocean Sciences Bowl: Extending the Reach of a High School Academic Competition to College, Careers, and a Lifelong Commitment to Science. American Secondary Education, 35(3).
Walters, Howard D., Bishop, Tina, & Wlodarsky, Rachel (2006). An Impact Assessment of the National Ocean Sciences Bowl—Lessons Learned from a National Two-Year Study. Special Report No.2: Current: The Journal of Marine Education. Published jointly by the Consortium of Oceanographic Research and Education, NOAA/National Ocean Services, and the National Marine Educators Association.
Evaluation Strategic Planning and Evaluation Advisory Support: The Joint Oceanographic Institution (now part of COL)
In 2005-2006, The College of Exploration (Dr. Walters, Dr. Bishop and Peter Tuddenham) conducted a strategic evaluation for the Joint Oceanographic Institution (JOI) of both existing JOI education tools and materials as well as an assessment of the broader ocean science and science education communities nationally with a view toward the continued development and enhancement of JOI's stature within those communities. This project led to the familiarization of TCOE evaluators with the JOI and IODP environment, culture, products and activities at that time. This has led to our current evaluation support project with former JOI staff (now part of COL), which is an NSF-funded informal science education project called Ship to Shore Science. http://www.oceanleadership.org/education/deep-earth-academy/educators/school-of-rock/sor-2012-ship-to-shore/
Our role in Ship to Shore Science is to advise and support the informal science education pilot projects and to inform and guide decision-making to the PIs as this project unfolds.
For this strategic evaluation, the evaluators supported the goal of comparing other nationally recognized successful education programs by establishing a project website with virtual meeting spaces for project personnel, establishing and conducting an online focus group of leading science educators, and preparing a report comparing JOI education program elements with comparable elements from similar national science education entities. To address the goal of both establishing baseline data and assessing the effectiveness of existing materials and scientific ocean drilling information against the baseline, Walters and Bishop developed and implemented a survey, which established a baseline of the extent of educators' use of the materials and methods of JOI, the degree to which the JOI materials and educational programs/methods met state, regional, and national science education standards and curricula frameworks, and the match between teacher instructional objectives. An analysis of focus group discussion comments and survey results helped identify strengths, weaknesses and opportunities for JOI Education. The evaluators also conducted a literature review of evaluation and educational methodologies, tool development, emerging trends in professional development for pre-service and in-service teachers, and teachers' use of the web as a source of curriculum materials. A final report provided a summary of all research and evaluative data with recommendations to support decision-making for JOI education and plans for future evaluation of materials and programs.
Online Workshop and Training Evaluation, Educator Needs Assessment
The College of Exploration has a distinguished national and international reputation for designing online spaces and programs for professional development having provided these for a broad cross set of agencies including: National Geographic Education, NOAA Office of Ocean Exploration and Research, and other NOAA Program Offices, including NOAA Education and NOAA Teacher at Sea, as well as various NSF national projects and partnerships. Using this online design, the College of Exploration further convened and facilitated the virtual planning processes that created the Ocean Literacy Essential Principles and Fundamental Concepts. Since 2000 the College of Exploration has regularly employed online surveys and online focus groups, as well as tracking online dialogue to assess the online professional development programs. In addition, The College of Exploration has provided needs assessment surveys for clients and for itself to help ascertain needs of educators for professional development and educational resources.
Bishop, Tina & Walters, Howard D. (2003). The Sustainable Seas Expedition: An Impact Assessment of an Online Teacher Education Program Funded by the National Geographic Society. Current: The Journal of Marine Education. Winter 2002.
Walters, Howard D. & Bishop, Tina (2004). The Classroom Exploration of the Oceans Online Workshop Series. Current: The Journal of Marine Education. June 2004.
Online Module Review and Lesson Plan Review
The College of Exploration has provided review and pilot-testing support to clients who have lesson plans and modules to test with educators. We have provided online focus groups for module review and also have created surveys to get feedback from lesson plan reviewers. Clients of this pilot testing include National Park Service, National Geographic Society, and NOAA Fisheries.
Throughout our time as evaluators, we have consistently and implicitly adhered to the principles of developmental evaluation, even when not explicitly articulating the approach. We have found that this approach, which was put forth by Michael Quinn Patton, offers a more holistic and creative view of evaluation. It has seemed appropriate for a number of our evaluation settings, in which goals are evolving and emerging as the project unfolds. Developmental evaluation realizes the complex and dynamic environment of educational programs and employs methods that engage the learning team in co-creating change. For a current project in New Hampshire--the Math and Science Partnership--The College of Exploration is working with the team to incorporate developmental evaluation principles as the project dynamically changes with new connections occurring in both STEM content and participants in districts across the state.