How to evaluate objective video quality metrics reliably

Jari Korhonen, Nino Burini, Junyong You, Ehsan Nadernejad

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review


The typical procedure for evaluating the performance of different objective quality metrics and indices involves comparisons between subjective quality ratings and the quality indices obtained using the objective metrics in question on the known video sequences. Several correlation indicators can be employed to assess how well the subjective ratings can be predicted from the objective values. In this paper, we give an overview of the potential sources for uncertainties and inaccuracies in such studies, related both to the method of comparison, possible inaccuracies in the subjective data, as well as processing of subjective data. We also suggest some general guidelines for researchers to make comparison studies of objective video quality metrics more reliable and useful for the practitioners in the field.
Original languageEnglish
Title of host publication2012 Fourth International Workshop on Quality of Multimedia Experience (QoMEX)
Publication date2012
ISBN (Print)978-1-4673-0724-6
ISBN (Electronic)978-1-4673-0725-3
Publication statusPublished - 2012
Event4th International Workshop onQuality of Multimedia Experience (QoMEX) - Melbourne, Australia
Duration: 5 Jul 20127 Jul 2012


Conference4th International Workshop onQuality of Multimedia Experience (QoMEX)
Internet address


  • Video quality assessment
  • Objective metrics
  • Reliability

Cite this