Gamification of student peer review in education - A systematic literature review

We present the first systematic review of the use of gamification in educational peer review activities. The goal of this work is to understand how gamification has been used to engage students in peer review activities and to summarize the empirical evidence for its effectiveness. Our main contribution is the presentation of a general model of the peer review process that captures the students’ activities and an examination of the specific actions within this model that have been gamified in the current literature. We also summarize the commonly used game mechanics and the context and year level of courses in which prior research has been conducted, along with the reported effects on student behavior. We find that artifact assessment and artifact creation are the two most commonly gamified actions with respect to our peer review model and that the quantity and quality of both the artifacts and the generated feedback are the most popular reward criteria. In addition, Science, Technology, Engineering and Mathematics (STEM) are the discipline areas in which gamified peer review activities are most often reported. In general, while the existing peer review literature reports mostly positive effects of gamification on student engagement, the range of student actions which have been incentivized remains narrow. Key activities, such as student reflection on the feedback received, have been largely unexplored with respect to gamification and thus present useful avenues for future work.

1. How have peer review activities been gamified?
   1. What actions have been gamified?
   2. What methods have been used to gamify the actions?
   3. What game elements have been used?
2. In which contexts (course/subject) and levels (year of study) have peer review been gamified?
3. What is the evidence and impact of the gamified peer review studies?
4. The goal of the first research question (RQ1) is to understand the various ways in which gamification has been used in peer review contexts. We split this broad question into three distinct aspects of the implementation: activities in peer review that have been gamified; methods to gamify peer review activities; game elements that have been used. The second research question (RQ2) focuses on identifying the context of the peer review, including the types of subjects/courses and the level of education of the students. The last research question (RQ3) examines the reported evidence for the impact of gamification. Together, this information will be valuable for understanding how the gamification of peer review activities can affect the learning process. (@indriasari2020a, 4)
5. 

1. 
   • I think this is quantitative since they're not doing any surveys? It's all about coding attributes from articles and analyzing the trends.
2. 
3. 
   • This section explains the “Gamified Peer Review Model” (Fig. 3) that we use in our analysis. It captures the activities that may occur during a student peer review process. The circular nodes represent the different roles that students adopt. The rectangular nodes symbolize objects. The arcs represent actions taken by students, mediated through the peer review system.

1. However, there are still relatively few studies that have investigated the gamification of peer review systems. (@indriasari2020a, 20)
2. More innovative implementations of gamification in peer review systems may also be the focus of future work. (@indriasari2020a, 21)
3. New systems that support the full range of interactions in the peer review model may be suitable for future research that looks to explore the application of gamification to the set of actions that have not yet been studied. (@indriasari2020a, 21)
4. However, systems such as Perusall (Miller et al. 2018) have demonstrated that machine learning techniques may be sufficient to provide automated grading of qualitative comments, suggesting that it may be possible to use such approaches to gamify qualitative feedback in the future. (@indriasari2020a, 21)
   • LLM to assess the quality of feedback must be so trivial now.
5. Further work that investigates the impact and trade-offs between automated and manual reward allocation would be beneficial. (@indriasari2020a, 21)
   • Like a Papers Please game, right?
6. Interestingly, although 19 studies were conducted in the area of STEM, only five of them reviewed source code (pure source code and mixed artifacts), despite review of source code being standard practice in industry (Li 2006; Sripada et al. 2015; Turner et al. 2008). Further research on the gamification of tasks involving peer review of source code is therefore warranted. (@indriasari2020a, 22)
   • Keep confirming me baby.
7. Reviewing artifacts and creating artifacts, the primary components of peer review, were the two most commonly gamified activities in the literature. (@indriasari2020a, 23)
8. The most popular game elements reported in the primary studies were points, badges, and leaderboards, mirroring the results of previous large-scale reviews of gamification. Most studies involved multiple game mechanics used in combination, and most were conducted in traditional or distance learning contexts in higher education settings. Interestingly, although most studies were conducted in computer science or software engineering courses, very few involved the review of source code. Future work on the gamification of peer review activities should look to target the gaps identified by this systematic review. (@indriasari2020a, 23)
   • Mine would also be for reviewing source code.

1. Key activities, such as student reflection on the feedback received, have been largely unexplored with respect to gamification and thus present useful avenues for future work. (@indriasari2020a, 1)
   • Another avenue to explore via the Papers Please game idea?