Master's Theses

Abstract:
Background and Context. Software testing is a fundamental component of computer science education, forming the basis for students’ ability to ensure program correctness and reliability. Despite its importance, many students struggle to design test cases that effectively expose faults and achieve meaningful test coverage. Traditional instructional approaches often emphasize code coverage metrics such as line or branch coverage, but these metrics may not adequately capture the quality of student tests. Mutation analysis, which measures how well tests detect small, artificial faults (mutants) introduced into the program, offers a potentially richer measure of test effectiveness. However, little is known about how feedback based on mutation coverage affects student learning compared to more conventional coverage-based feedback.
Objectives. This thesis investigates how different forms of automated feedback impact students’ underlying ability to construct effective test suites. Specifically, it examines whether exposure to mutation-based feedback or branch coverage feedback better improves students’ conceptual understanding and independent application of software testing principles. In other words, rather than asking which feedback produces stronger test suites during training, this study asks which feedback better helps students learn to become stronger testers.
Method.To investigate these questions, a controlled experiment was conducted with students in undergraduate software engineering courses. Participants were randomly assigned to one of two treatment groups: one receiving automated feedback based on branch coverage, and the other receiving feedback based on mutation coverage. The study employed a pre/post-test design to measure learning gains independent of feedback exposure. In the beginning, all students completed a set of baseline (pre-test) problems in which they wrote test suites without receiving any feedback. During the intervention phase, each group practiced test writing while receiving their respective feedback type. Afterward, students completed a new set of problems (post-test) again without feedback, allowing direct measurement of how much their testing ability improved as a result of the practice sessions. Data from all three phases (pre-test, post-test, practice) was analyzed to assess changes in the correctness and completeness of the test suites, as well as additional statistical characteristics such as the quantity and redundancy of the test cases.
Findings. The results showed that students who received mutation coverage feedback demonstrated a greater improvement in test-writing ability between the pre- and post-tests compared to those who received branch coverage feedback. Specifically, the mutation feedback group achieved a higher increase in fault detection and completeness, as well as a larger gain in mutation coverage from pre- to post-tests. During the practice phase, students receiving mutation feedback also wrote more tests on average than those receiving branch feedback, reflecting a more exploratory and fault-oriented testing approach. These outcomes suggest that exposure to mutation-based feedback encourages deeper reasoning about program behavior and more persistent refinement of test suites, leading to stronger transfer of testing skills beyond the feedback environment.
Implications. Although results remain preliminary, this work contributes to the growing body of research on assessment and feedback in computing education by examining how different adequacy metrics can shape students’ testing practices. The findings are expected to inform instructional design decisions regarding the integration of advanced testing metrics, such as mutation analysis, into computer science curricula. Ultimately, this research aims to provide actionable insights for educators seeking to improve how software testing concepts are taught and assessed in undergraduate computer science programs.

Abstract: Peers are an invaluable resource for students at undergraduate universities. Many factors can impact how students form connections, some of which are tied to the students’ identities. While social networks have been studied in the context of universities, little research has been done specifically about peer collaboration and even less in the context of computer science classes. Our research aims to gain an understanding of how peer networks form in computing classes and the effect of being involved in one on students’ academic performances. We collected survey data (n = 139) about students’ peer collaborative behaviors in computer science classes. Several patterns were observed. Students were more likely to collaborate with students with their same demographic identity, including gender and ethnicity. A student’s demographic identity had no predictive quality on their participation in a peer network. Students who participated in a peer network had a higher grade performance in the class compared to students who did not.

Abstract: Open-source software (OSS) ecosystems, defined as environments composed of package managers and programming languages (e.g., NPM for JavaScript), are essential for software development and foster collaboration and innovation. Although their significance is acknowledged, understanding what makes OSS communities healthy and sustainable requires further exploration. This thesis quantitatively assesses the health of OSS projects and communities within the NPM, PyPI, Maven, and RubyGems ecosystems. We explore five research questions addressing project standards, community responsiveness, contribution distribution, contributor retention, and newcomer integration strategies. Our analysis shows varied documentation practices, insider engagement levels, and contribution patterns. Our findings highlight both strengths and different areas for improvement across ecosystems. For example, RubyGems excels in the adoption of project documentation and exhibits the most even distribution of contributions among all contributors, including highly active contributors. and a very responsive community, but it needs to improve contribution retention and attract newcomers to the projects. Meanwhile, NPM and Maven show a trend toward getting new contributors, characterized by a high ratio of individual contributions. They need to better adopt a code of conduct, pull request templates, and increase the number of active contributors in a project. This thesis offers insights to developers and maintainers on how to strengthen ecosystems and support vibrant communities effectively.

Abstract: The ability to test software is invaluable in all areas of computer science, but it is often neglected in computer science curricula. Test adequacy criteria (TAC), tools that measure the effectiveness of a test suite, have been used as aids to improve software testing teaching practices, but little is known about how students respond to them. Studies have examined the cognitive processes of students programming and professional developers writing tests, but none have investigated how student testers test with TAC. If we are to improve how they are used in the classroom, we must start by understanding the different ways that they affect students' thought processes as they write tests. In this thesis, we take a grounded theory approach to reveal the underlying cognitive processes that students utilize as they test under no feedback, condition coverage, and mutation analysis. We recorded 12 students as they thought aloud while creating test suites under these feedback mechanisms, and then we analyzed these recordings to identify the thought processes they used. We present our findings in the form of the phenomena we identified, which can be further investigated to shed more light on how different TAC affect students as they write tests.

Abstract: Programming is a key skill in many domains outside computer science. When used judiciously, programming can empower people to accomplish what might be impossible or difficult with traditional methods. Unfortunately, students, especially non-CS majors, frequently have trouble while learning to program. This work reports on the challenges and opportunities faced by Physical Chemistry (PChem) students at Cal Poly, SLO as they learn to program in MATLAB. We assessed the PChem students through a multiple-choice concept inventory, as well as through “think-aloud” interviews. Additionally, we examined the students' perceptions of and attitudes towards programming. We found that PChem students are adept at applying programming to a subset of problems, but their knowledge is fragile; like many intro CS students, they struggle to transfer their knowledge to different contexts and often express misconceptions about programming. However, they differ in that the PChem students are first and foremost Chemistry students, and so struggle to recognize appropriate applications of programming without scaffolding. Further, many students do not perceive themselves as competent general- purpose programmers. These factors combine to discourage students from applying programming to novel problems, even though it may be greatly beneficial to them. We leveraged this data to create a workshop with the goal of helping PChem students recognize their programming knowledge as a tool that they can apply to various contexts. This thesis presents a framework for addressing challenges and providing opportunities in domain-specific CS education.

Abstract: Knowing when and how to seek academic help is crucial to the success of undergraduate computing students. While individual help-seeking resources have been studied, little is understood about the factors influencing students to use or avoid certain resources. Understanding students' patterns of help-seeking can help identify factors contributing to utilization or avoidance of help resources by different groups, an important step toward improving the quality and accessibility of resources. We present a mixed-methods study investigating the help-seeking behavior of undergraduate computing students. We collected survey data (n=138) about students' frequency of using several resources followed by one-on-one student interviews (n=15) to better understand why they use those resources. Several notable patterns were found. Women sought help in office hours more frequently than men did and computing majors sought help from their peers more often than non-computing majors. Additionally, interview data revealed a common progression in which students started from easily accessible but low utility resources (online sources and peers) before moving on to less easily accessible, high utility resources (like instructor office hours). Finally, while no differences between racial groups was observed, the lack of diversity in our sample limits these findings.

Abstract: As the demand for programming skills in today's job market is rapidly increasing for disciplines outside of computing, CS courses have experienced spikes in enrollment for non-majors. Students in disciplines including art, design and biological sciences are now often required to take introductory CS courses. Previous research has shown the role of growth mindset, self-efficacy and relevance in student success within CS but such metrics are largely unknown for non-majors. In this thesis, we surveyed non-majors in CS1 at Cal Poly, San Luis Obispo during the early and late weeks of the quarter to gain insights on their growth mindset, their self-efficacy and the perceived relevance of the course to their lives. In our analysis, we discovered that non-majors' levels of growth mindset and of self-efficacy decreased throughout the duration of CS1 with additional differences by gender. However, non-majors largely found that the material covered in CS1 was highly relevant to their academic and professional careers despite being challenged by it. These findings provide important insights into the experiences of non-majors learning to code and can help better serve a more diverse population of students.