Optimal allocation of reviewers for peer feedback

David Kofoed Wind, Ulf Aslak Jensen, Rasmus Malthe Jørgensen, Simon Lind Hansen, Ole Winther

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review


Peer feedback is the act of letting students give feedback to each other on submitted work. There are multiple reasons to use peer feedback, including students getting more feedback, time saving for teachers and increased learning by letting students reflect on work by others. In order for peer feedback to be effective students should give and receive useful feedback. A key challenge in peer feedback is allocating the feedback givers in a good way. It is important that reviewers are allocated to submissions such that the feedback distribution is fair - meaning that all students receive good feedback.

In this paper we present a novel way to intelligently allocate reviewers for peer feedback. We train a statistical model to infer the quality of feedback based on a dataset of feedback quality evaluations. This dataset contains more than 20,000 reviews where the receiver of the feedback has indicated the quality of the feedback. Using this model together with historical data we calculate the feedback-giving skill of each student and uses that as input to an allocation algorithm that assigns submissions to reviewers, in order to optimize the feedback quality for all students.

We test the performance of our allocation strategy using real data from over 600 peer feedback sessions and simulate the effects of different allocation strategies. By comparing our method with a random allocation algorithm and a “super-informed oracle” algorithm we demonstrate that we are able to allocate reviewers to submissions in such a way that all submissions receive feedback of similar quality and that we are able to significantly outperform simple random allocation of reviewers. Additionally we investigate the effect of pre-allocating reviews in comparison to allocating reviewers live during the review process and show that live-allocation leads to better results. Our method is robust to reviews not being completed and other real-life quirks and improves as more feedback data is collected.
Original languageEnglish
Title of host publicationProceedings of 16th European Conference on eLearning
PublisherAcademic Conferences and Publishing International
Publication date2017
ISBN (Print)978-1-911218-60-9
Publication statusPublished - 2017
Event16th European Conference on eLearning - Porto, Portugal
Duration: 26 Oct 201727 Oct 2017


Conference16th European Conference on eLearning

Bibliographical note

For ECEL2017 http://www.academic-conferences.org/conferences/ecel/


  • peer assessment
  • peer feedback
  • feedback
  • peer review
  • peer evaluation
  • peer grading
  • task assignment
  • reviewer allocation


Dive into the research topics of 'Optimal allocation of reviewers for peer feedback'. Together they form a unique fingerprint.

Cite this