How Computer Science Students Learn to Debug

How Computer Science Students Learn to Debug: A Literature Review

When compared to the timeline of more traditional disciplines, the study of Computer Science is quite young. Therefore, one may prematurely conclude that the body of research around how Computer Science students learn is relatively small - especially compared to other disciplines that have existed for centuries prior. Luckily, this is not the case, and there exists a large body of research studying various aspects of learning in Computer Science including learning how to write programs and debug them.

It is virtually impossible to write a computer program that does not contain any bugs, with the first (physical) kind dating back to 1946 (Marinissen, 2008, p. 288). The act of debugging, or removing defects from a program, requires the application of cognitive processes such as analysis, evaluation, and problem-solving. The application of these processes may happen on an individual basis or collaboratively in a group. The internet is an invaluable resource when debugging. However, the ability to filter useful information from the ocean of concepts and ideas one is flooded with when performing a search query is required. Distinguishing between what one knows from what one does not know (i.e., metacognition) becomes an invaluable skill. The following literature review explores the cognitive processes required by Computer Science students to problem-solve (individually and collaboratively) using the internet to debug their code.

Debugging

@chmiel2004 found that novice programmers could not create a mental model and have a narrow scope of vision when debugging. Students were found to frequently jump to conclusions based on the first thing they saw instead of understanding the problem in its entirety. On the other hand, skilled programmers were adept at building a mental model of the program by analyzing the relationships between different areas to deduce potential problems better. The authors observed that novice programmers did not come up with hypotheses about where the problem could be coming from. Code reviews are an effective technique to identify and fix bugs, and novice programmers can learn much from their peers in this process (Chmiel & Loui, 2004, p. 17; Fitzgerald et al., 2010; Marques, Ochoa, Bastarrica, & Gutierrez, 2018; Murphy-Hill, 2014). @chmiel2004 designed activities to improve students' debugging capabilities at the University of Illinois in ECE 291, which all computer engineering and electrical engineering students must take. These activities were designed using strategies found in the literature. The study was conducted in the Spring of 2003, and it involved 27 students.

Students were given code that contained bugs. They were tasked with analyzing the code during a code review process to identify, but not fix, as many bugs as they could find in the code. Identification of the bug was the first step in the debugging process. Then, the students were given different programs first to identify the bugs and then fix the bugs. The students had access to a debugging tool to step through the code, set breakpoints, and read the values of all the variables. During this process, the students were tasked with keeping a log of all the bugs. They had to write down information such as which function the bug was coming from, how long it took them to fix, and the desired output vs the actual output of the program and the fix itself. This log could then be referred to later if they ever encountered a similar bug. @chmiel2004 [p. 19] found significant results regarding the time spent debugging by students who had done the debugging exercises versus those who had not. Debugging engages the problem-solving areas of a student's brain and must be practiced repeatedly to get better at it (Chmiel & Loui, 2004, p. 21).

To begin looking at a student's brain, one must examine the student's eyes. @lin2016 studied students' ability to debug a program using an eye-tracking device effectively. This study aimed to determine the cognitive processes (mainly problem-solving) used by both high and low-performing students while debugging. The authors define a beacon as a potential area in the code that might be causing the bug. More experienced programmers are faster at locating these beacons. Eye-tracking was used to determine the speed and accuracy of identifying beacons in both high and low-performing students. The experiment was done on 22 male and 16 female undergraduate Computer Science students who had at least one year of experience in the C programming language. @lin2016 ultimately found that the low-performing students were aimless and could not develop a debugging plan of attack. On the other hand, high-performing students could engage in many cognitive processes to understand the problem and build a solid mental model of the program to triangulate the bug's location.

Debugging is challenging for novice programmers since it requires several other cognitive processes to perform effectively (Fitzgerald et al., 2010). Students must know programming fundamentals, read code, understand the different instructions, and have a general idea of where the problems or issues could be coming from. They must also be able to recall information, identify the correct context of the problem, and use programming constructs correctly. To identify gaps in their knowledge, @fitzgerald2010 [p. 390] hoped to analyze the differences between what students said they did versus what they did. The authors identified that much of the research about the debugging skills of students was done in the 1980s. Their study aimed to utilize strategies from those older studies using modern programming environments during the Spring of 2007 at six universities in the United States and one university in the United Kingdom. The study involved students doing a programming exercise, debugging the program, participating in an interview, and filling out a debugging survey (Fitzgerald et al., 2010, p. 391). The authors report that debugging requires strong problem-solving skills and strategies and that debugging should be explicitly taught by teachers. The students should be tasked with thinking about all the different ways the error could be happening before allowing them to alter the code. This analysis would involve going back to the problem statement to precisely nail down the desired output. Students must practice articulating between "desired output" and "actual output". To reduce cognitive overload, teachers should take care to only use one programming style (e.g., naming conventions, brace placement) (Fitzgerald et al., 2010, p. 395). Software testing and debugging are complimentary practices. Teaching the students about good testing practices through live testing demonstrations can enable students to think about the desired results (Fitzgerald et al., 2010, p. 395). The authors posit that using the internet to search for solutions should be discouraged for extremely novice programmers since these students can easily get lost in the plethora of information and resources on the web. They conclude by stating that more research should be done about the novice programmer's use of the internet to solve problems (Fitzgerald et al., 2010, p. 396).

Searching the Internet

For instance, searching the internet for reliable sources of information to solve problems is difficult for Computer Science students (Ben-David Kolikant & ma’ayan, 2018, p. 213). Countless authors' sheer quantity of documentation makes it hard to filter out irrelevant information, even for experienced programmers. @ben-davidkolikant2018 interviewed twelve undergraduate Computer Science students to find out how the students felt about performing searches while programming. The objectives were to identify common areas of difficulty when searching and the cognitive processes involved in using the internet to solve programming bugs. Of the twelve students, six were from Britain and six from Israel. Each group of six contained half male and half female students. Since the discipline is heavily male-dominated, it is good that they considered this disparity in their research. @ben-davidkolikant2018 [p. 215] found that first-year students did not use relevant words about their current problem when phrasing their queries. The first-year students tended to take whatever the first search engine result was and copy-paste it in their code without knowing what it did. None of the interviewees stated that they had any formal training in how to use search engines effectively. Knowing the right questions to ask is a byproduct of increased experience (Ben-David Kolikant & ma’ayan, 2018, p. 220). The authors concluded by asking if teachers can aid students' problem-solving skills by teaching them how to search effectively.

Problem-Solving

Debugging is not the only place where students need to apply problem-solving skills. @soh2015 [p. 33] illustrate how Computer Science students not only need to know how to think computationally but think creatively as well. Computational thinking and creative thinking should not be seen as a dichotomy. The authors used Epstein's Generativity Theory (Epstein, 1993) as a framework for the tool they developed to help students with computational and creative thinking. The framework consists of 4 dimensions:

1. Broadening: One can tackle a more comprehensive array of problems if one possesses skills from many domains.
2. Challenging: When solutions one uses typically do not work, one can develop a more creative solution.
3. Surrounding: While broadening applies to diverse knowledge and skills (how/what), surrounding applies to diverse environments (when/where/who). The more people/situations one interacts with will lead to experiencing new perspectives one may not have considered before.
4. Capturing: The act of purposefully recording new solutions to problems and new skills as one learns them to use them in future situations more effectively.

@soh2015 [p. 33] created Computational Creativity Exercises (CCEs) using Epstein's Generativity Theory principles. For example, describe an everyday object to someone who has never heard of it (e.g., mechanical pencil). Analyze the object and determine how to use it in code (i.e., class/function/variable). Reflect on the solution and determine if any functions can be abstracted to fit a more general purpose. In addition to creative and computational thinking, the CCE's help the students by getting them to create connections between ideas and concepts. Through repetition, the students will learn through using discipline-specific language at higher levels of abstraction. @soh2015 found a significant correlation between the rate of completion of CCE's and the students' grades in the course. A computational thinking knowledge test was administered to gauge the effectiveness of the CCEs, and the results were positive. The more CCEs a student completed, the higher their letter grade was and the higher they scored on the computational thinking knowledge test, which, among other cognitive processes, required both problem-solving and critical skills (Soh, Shell, Ingraham, Ramsay, & Moore, 2015, p. 35).

@fee2010 also identified problem-solving and critical thinking as two of the most significant cognitive processes needed by Computer Science students. They defined Problem-Based Learning (PBL) as a learning approach that puts students in the driver's seat and constructs their learning. PBL was done through scaffolding the problem-solving process such that the students could pull from prior learning (Ben-David Kolikant & ma’ayan, 2018, p. 219) to tackle new problems. The authors stated that the student must be an active part of the process (Fee & Holland-Minkley, 2010, p. 130) and noticed that students do not realize that the core of any Computer Science related goal is problem-solving. @fee2010 state that this is a challenge when implementing PBL since the students tend to be more motivated by the end product and not the process or journey to get there. All 52 Computer Science students at Washington & Jefferson College over five years in the final capstone project graduated. The authors view this as an indication that the PBL approach is ultimately a successful one, though they admit that the sample size is relatively small. Further research, including interviewing past graduates, could be helpful for this topic. This article is an essential example of students solving problems on their own.

Collaboration

Conversely to solving problems on one's own, @murphy-hill2014 states that humans have most effectively learned through social interactions. The author wants to explore how much they can leverage social learning with the advances in modern technology. In Computer Science, this takes the shape of pair programming where two students sit together at one computer and work collaboratively on one problem. Pair programming can also be done synchronously online with what the author refers to as continuous screencasting. @murphy-hill2014 posits nine principles of effective social learning: recording efficiency, learning efficiency, privacy preservation, targeting, trust, rationale, feedback efficiency, bootstrapping, and generality. Sites like Stack Overflow and GitHub are built with social interactivity and learning at their core. There are many reasons why developers might hesitate to use new tools, but @murphy-hill2014 states that the discoverability of new tools and how to use those tools can be solved by turning to one's peers.

One developer can learn from another simply by engaging in a pair programming session with them (Fitzgerald et al., 2010, p. 395; Murphy-Hill, 2014). By explaining their thought process out loud to their peer, they are forced to build a mental model of how the code is being executed. Speaking with peers also exposes the student to a variety of approaches. This collaboration also builds trust in the team, which can lead to an enhanced learning experience. For example, the students would work individually on a program and then collaborate with peers to identify potential bugs before executing the program. Then they would work together to solve any bugs after the program execution (Chmiel & Loui, 2004, p. 18).

@marques2018 also studied the learning effects of collaboration on Software Engineering students. They used a formative monitoring tool called Reflexive Weekly Monitoring (RWM) while doing team projects to see how much students were aware of their effectiveness on the team. Using this RWM tool, the authors hoped that the students would employ self-reflective and collaborative practices to monitor if their contributions to the team were valuable. The reflection process had students question the team's main weaknesses and come up with answers to why the team was behind schedule. The nine-semester-long study took place with a total of 205 students divided into thirty-two teams. The teams had to follow agile software development practices, which is common in the industry. The study results concluded that using RWM is effective and that having students engage in the cognitive processes of collaboration and self-reflection almost always leads to a more successful product. Marques et al. (2018) finish the article by acknowledging that more research is to be done in this area to follow each student's productivity closely. In the end, groups that did not use the RWM had trouble identifying the gaps in the knowledge in the team.

Metacognition

Being aware of what one knows and the gaps in one's knowledge can be referred to as metacognition and is a characteristic of an expert programmer (Bransford, Brown, & Pellegrino, 2000, p. 47). The metacognitive process requires reflection on what solutions one discovered and whether any functions can be abstracted to fit a more general-purpose context (Bransford, Brown, & Pellegrino, 2000, p. 34). In the studies by @chmiel2004 [p. 18] and @fitzgerald2010 [p. 395] about developing debugging skills for Computer Science students, they tasked students with answering metacognitive questions such as:

• How much time did you spend on designing, coding, and testing each part or subroutine?
• What kinds of defects did you find during the development of the program?
• When did you discover these defects (during code review or testing)?
• How did you find them?
• What would you do differently for the next programming assignment?
• Why did you make that change?
• What do you think will happen now?

By asking these questions, students can develop the ability to teach themselves (Bransford, Brown, & Pellegrino, 2000, p. 50). Metacognition also surfaced in the studies by @marques2018 and @soh2015, where they identified that novices are unable to distinguish between what they know and what they do not know. In addition, @ben-davidkolikant2018 [p. 220] outlined in their study that not only are the fundamental cognitive processes for searching the internet analysis and evaluation, but also metacognition. The students first had to recognize that the solution they were currently looking at was not one they understood, before doing some more searching to understand further how the solution worked.

Conclusion

The debugging, problem-solving, and metacognition skills required by Computer Science students to be effective at debugging have been studied thoroughly in the literature. Searching the internet for solutions has been identified as a practice that can simultaneously help and hinder a student's journey to finding a solution for their bug. If students work together collaboratively, they may help one another by providing perspectives and insights the others did not initially see. However, these insights are ultimately fleeting if the student does not take the time to self-reflect on the process and identify the gaps in their knowledge, what went right, and what could have gone better. Many of the authors have stated that further research is required to study these topics under different contexts and circumstances more closely. That being said, the studies that have been conducted so far about learning in Computer Science are excellent resources that any teacher or student could benefit from greatly.

References