Evaluation and Educational Research
The College of Exploration has provided evaluation leadership and support to a variety of STEM education projects at the national level over the past 25 years. Our experience and expertise include (but are not limited to):
- Longitudinal Research
- Needs Assessment
- Program Evaluation
- Project Evaluation Guidance
- Advisory and Technical Support
- Survey Design
- Lesson Plan and Educational Module Review
- Focus Group implementation
- Developmental Evaluation
Examples of Projects
NOAA Living Marine Resources Cooperative Science Center https://wwwcp.umes.edu/lmrcsc/
The College of Exploration is now engaged through a second round of funding (2022–2025) for external evaluation of the Living Marine Resources Cooperative Science Center (LMRCSC). This Center, funded by NOAA’s Educational Partnership Program for Minority Serving Institutions, is led by the University of Maryland Eastern Shore and is a consortium of seven universities nationwide who collaborate on the educational programming for Fisheries and other NOAA mission priorities.
The Center “trains and graduates students from underrepresented communities in marine science for careers in research, management, and public policy that support the sustainable harvest and conservation of our nation’s living marine resources”. With its partner institutions, the LMRCSC conducts research on marine and estuarine systems consistent with NOAA science. It offers courses, workshops, internships and other educational opportunities for students at the undergraduate, graduate, and post- doctoral levels, as well as teacher and faculty support for teaching and learning about fisheries, ecosystem management, and aquaculture.
The evaluation team, working with the LMRCSC team utilizes multiple sources of information, including site visits to inform the impacts of project members and activities toward the identified outputs and outcomes, the impact of the project on the members themselves as social network participants, and the broader impacts of the project on the various participants and beneficiaries of the project activities. A main focus is on looking at benefits to students at all levels from their LMRCSC activities and tracking their career pathways as they go through the LMSCRC system and out into the working world.
In Search of Earth’s Secrets
The College of Exploration provided external evaluation services for the In Search of Earth’ Secrets five-year project funded by the AISL program of NSF. The goal of this project was to make deep earth/ocean science available to informal audiences, bringing the excitement of the Joides Resolution ship and its underwater scientific ocean drilling expeditions to the general public-learners of all ages at libraries, museums, and science centers.
The team, under the direction of Consortium of Ocean Leadership and Lamont-Doherty Earth Observatory consisted of informal science venues personnel, scientists and educators in the JRON community, exhibit and graphic designers, Girl Scout groups as well as an evaluation team and an advisory board.
The evaluators looked at the learning outcomes of the project for all participants and stakeholders, with special emphasis placed on minorities, rural populations and girls through the Girl Scout organization. The evaluation examined the use of STEM content and skills development in public outreach, network and partnership support for learning, support for informal educational venues and enhanced interest and awareness in ocean science by the public.
Integration and Application (IAN) Network
ian.umces.edu
The Integration and Application Network (IAN) of the University of Maryland Center for Environmental Science (UMCES) has engaged the services of the College of Exploration over the past 20 years on a variety of strategic initiatives relating to the environment including report cards, organizational development, and evaluation. projects.
IAN creates innovative ways to visually present science, making it more accessible to managers, stakeholders, and the general public. IAN offers science communication services, environmental report card production, and training and capacity building. The College of Exploration worked with the IAN team developing and implementing an organizational assessment process.
The College of Exploration has also provided support for the IAN group for professional development, looking at communication approaches, evaluation methods, enrichment activities, and team sharing as a learning system.
The College of Exploration also provided evaluation guidance and strategic support for UMCES IAN’s project The Coastal Ocean Assessment for Sustainability and Transformation (COAST Card), which is a collaborative research program funded by the Belmont Forum. It brings together researchers from the United States, Philippines, Norway, India, and Japan. The COAST Card merges three essential approaches: stakeholders identification and societal guidance, status assessment, and prioritized linked actions. Through a transdisciplinary and transnational approach, the project engages stakeholders in a co-design and co-development process, to foster solutions to global sustainability challenges.
The National Ocean Sciences Bowl
www.nosb.org
One long-term project is a longitudinal research study for the Consortium for Ocean Leadership’s National Ocean Sciences Bowl since 1999. This study has looked holistically at the entire NOSB program and its effects on a variety of key audiences, but in particular it follows student alumni who participate in the competition. One key goal has been to track these alumni participants as they move into college and then on to careers. The College of Exploration has managed to keep in touch with more than 500 alumni who regularly provided our team with information about their college choices, coursework, majors and career choices.
For this NOSB research study TCOE has developed dozens of survey instruments and multiple interview protocols to guide project development. Multiple juried publications have emerged from this body of work including:
Walters, H., Bishop, K. (2019). Systematic and Longitudinal Research to Support and Improve the National Ocean Sciences Bowl: Findings from Seventeen Years of Study. In Fauville, G., Payne, D., Marrero, M., Lantz-Andersson, A., Crouch, F. (eds) Exemplary Practices in Marine Science Education. Springer, Cham. https://doi.org/10.1007/978-3-319-90778-9_5
Bishop, Tina & Walters, Howard D. (2007). The National Ocean Sciences Bowl: Extending the Reach of a High School Academic Competition to College, Careers, and a Lifelong Commitment to Science. American Secondary Education, 35(3).
Walters, Howard D., Bishop, Tina, & Wlodarsky, Rachel (2006). An Impact Assessment of the National Ocean Sciences Bowl—Lessons Learned from a National Two-Year Study. Special Report No.2: Current: The Journal of Marine Education. Published jointly by the Consortium of Oceanographic Research and Education, NOAA/National Ocean Services, and the National Marine Educators Association.
Assessing the Impact of the National Ocean Sciences Bowl: A Systems Approach
The College of Exploration is now working with the NOSB to help evaluate a set of online resources, which focus on providing quiz and activities for teachers’ in their everyday classroom use. These online resources incorporate NOSB questions, videos, activities and other curriculum support. These resources are situated on the Scoutlier online system, which positions itself as the ultimate digital teaching assistant, which helps streamline, enhance, and customize lessons.
Evaluation Strategic Planning and Evaluation Advisory Support: The Joint Oceanographic Institution
In 2005-2006, The College of Exploration conducted a strategic evaluation for the Joint Oceanographic Institution (JOI) of both existing JOI education tools and materials as well as an assessment of the broader ocean science and science education communities nationally with a view toward the continued development and enhancement of JOI’s stature within those communities. This project led to the familiarization of TCOE evaluators with the JOI and IODP environment, culture, products and activities at that time. This led to our later evaluation support project with former JOI staff, which was an NSF-funded informal science education project called Ship to Shore Science. Our role in Ship to Shore Science was to advise and support the informal science education pilot projects and to inform and guide decision-making to the PIs as the project unfolded. An analysis of focus group discussion comments and survey results helped identify strengths, weaknesses and opportunities for JOI Education.
For this strategic evaluation, the evaluators supported the goal of comparing other nationally recognized successful education programs by establishing a project website with virtual meeting spaces for project personnel, establishing and conducting an online focus group of leading science educators, and preparing a report comparing JOI education program elements with comparable elements from similar national science education entities. To address the goal of both establishing baseline data and assessing the effectiveness of existing materials and scientific ocean drilling information against the baseline, the College of Exploration developed and implemented a survey, which established a baseline of the extent of educators’ use of the materials and methods of JOI, the degree to which the JOI materials and educational programs/methods met state, regional, and national science education standards and curricula frameworks, and the match between teacher instructional objectives.
Online Workshop and Training Evaluation, Educator Needs Assessment
The College of Exploration has a distinguished national and international reputation for designing online spaces and programs for professional development having provided these for a broad cross set of agencies including National Geographic Education, NOAA Office of Ocean Exploration and Research, and other NOAA Program Offices, including NOAA Education and NOAA Teacher at Sea, as well as various NSF national projects and partnerships. Using this online design, the College of Exploration further convened and facilitated the virtual planning processes that created the Ocean Literacy Essential Principles and Fundamental Concepts. Since 2000 the College of Exploration has regularly employed online surveys and online focus groups, as well as tracking online dialogue to assess the online professional development programs. In addition, The College of Exploration has provided needs assessment surveys for clients and for itself to help ascertain needs of educators for professional development and educational resources.
Bishop, Tina & Walters, Howard D. (2003). The Sustainable Seas Expedition: An Impact Assessment of an Online Teacher Education Program Funded by the National Geographic Society. Current: The Journal of Marine Education Winter 2002.
Walters, Howard D. & Bishop, Tina (2004). The Classroom Exploration of the Oceans Online Workshop Series. Current: The Journal of Marine Education. June 2004.
Online Module Review and Lesson Plan Review
The College of Exploration has provided review and pilot-testing support to clients who have lesson plans and modules to test with educators. We have provided online focus groups for module review and also have created surveys to get feedback from lesson plan reviewers. Clients of this pilot testing include National Park Service, National Geographic Society, and NOAA Fisheries.
Developmental Evaluation
Throughout our time as evaluators, we have consistently and implicitly adhered to the principles of developmental evaluation, even when not explicitly articulating the approach. We have found that this approach, which was put forth by Michael Quinn Patton, offers a more holistic and creative view of evaluation. It has seemed appropriate for several of our evaluation settings, in which goals are evolving and emerging as the project unfolds. Developmental evaluation realizes the complex and dynamic environment of educational programs and employs methods that engage the learning team in co-creating change. For a prior project in New Hampshire–the Math and Science Partnership–The College of Exploration worked with the team to incorporate developmental evaluation principles as the project dynamically changed with new connections occurring in both STEM content and participants in districts across the state.