How to evaluate objective video quality metrics reliably

Jari Korhonen; Nino Burini; Junyong You; Ehsan Nadernejad

doi:10.1109/QoMEX.2012.6263839

How to evaluate objective video quality metrics reliably

Jari Korhonen
, Nino Burini
, Junyong You
, Ehsan Nadernejad

Norwegian University of Science and Technology

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

Abstract

The typical procedure for evaluating the performance of different objective quality metrics and indices involves comparisons between subjective quality ratings and the quality indices obtained using the objective metrics in question on the known video sequences. Several correlation indicators can be employed to assess how well the subjective ratings can be predicted from the objective values. In this paper, we give an overview of the potential sources for uncertainties and inaccuracies in such studies, related both to the method of comparison, possible inaccuracies in the subjective data, as well as processing of subjective data. We also suggest some general guidelines for researchers to make comparison studies of objective video quality metrics more reliable and useful for the practitioners in the field.

Original language	English
Title of host publication	2012 Fourth International Workshop on Quality of Multimedia Experience (QoMEX)
Publisher	IEEE
Publication date	2012
Pages	57-62
ISBN (Print)	978-1-4673-0724-6
ISBN (Electronic)	978-1-4673-0725-3
DOIs	https://doi.org/10.1109/QoMEX.2012.6263839
Publication status	Published - 2012
Event	4th International Workshop onQuality of Multimedia Experience (QoMEX) - Melbourne, Australia Duration: 5 Jul 2012 → 7 Jul 2012 http://www.qomex2012.org/

Conference

Conference	4th International Workshop onQuality of Multimedia Experience (QoMEX)
Country/Territory	Australia
City	Melbourne
Period	05/07/2012 → 07/07/2012
Internet address	http://www.qomex2012.org/

Keywords

Video quality assessment
Objective metrics
Reliability

Access to Document

10.1109/QoMEX.2012.6263839

OpenUrl availability

Check availability in DTU Findit

Cite this

@inproceedings{1a76eed93f5f4eb68c7531006175d30c,

title = "How to evaluate objective video quality metrics reliably",

abstract = "The typical procedure for evaluating the performance of different objective quality metrics and indices involves comparisons between subjective quality ratings and the quality indices obtained using the objective metrics in question on the known video sequences. Several correlation indicators can be employed to assess how well the subjective ratings can be predicted from the objective values. In this paper, we give an overview of the potential sources for uncertainties and inaccuracies in such studies, related both to the method of comparison, possible inaccuracies in the subjective data, as well as processing of subjective data. We also suggest some general guidelines for researchers to make comparison studies of objective video quality metrics more reliable and useful for the practitioners in the field.",

keywords = "Video quality assessment, Objective metrics, Reliability",

author = "Jari Korhonen and Nino Burini and Junyong You and Ehsan Nadernejad",

year = "2012",

doi = "10.1109/QoMEX.2012.6263839",

language = "English",

isbn = "978-1-4673-0724-6",

pages = "57--62 ",

booktitle = "2012 Fourth International Workshop on Quality of Multimedia Experience (QoMEX)",

publisher = "IEEE",

address = "United States",

note = "4th International Workshop onQuality of Multimedia Experience (QoMEX) ; Conference date: 05-07-2012 Through 07-07-2012",

url = "http://www.qomex2012.org/",

}

TY - GEN

T1 - How to evaluate objective video quality metrics reliably

AU - Korhonen, Jari

AU - Burini, Nino

AU - You, Junyong

AU - Nadernejad, Ehsan

PY - 2012

Y1 - 2012

N2 - The typical procedure for evaluating the performance of different objective quality metrics and indices involves comparisons between subjective quality ratings and the quality indices obtained using the objective metrics in question on the known video sequences. Several correlation indicators can be employed to assess how well the subjective ratings can be predicted from the objective values. In this paper, we give an overview of the potential sources for uncertainties and inaccuracies in such studies, related both to the method of comparison, possible inaccuracies in the subjective data, as well as processing of subjective data. We also suggest some general guidelines for researchers to make comparison studies of objective video quality metrics more reliable and useful for the practitioners in the field.

AB - The typical procedure for evaluating the performance of different objective quality metrics and indices involves comparisons between subjective quality ratings and the quality indices obtained using the objective metrics in question on the known video sequences. Several correlation indicators can be employed to assess how well the subjective ratings can be predicted from the objective values. In this paper, we give an overview of the potential sources for uncertainties and inaccuracies in such studies, related both to the method of comparison, possible inaccuracies in the subjective data, as well as processing of subjective data. We also suggest some general guidelines for researchers to make comparison studies of objective video quality metrics more reliable and useful for the practitioners in the field.

KW - Video quality assessment

KW - Objective metrics

KW - Reliability

U2 - 10.1109/QoMEX.2012.6263839

DO - 10.1109/QoMEX.2012.6263839

M3 - Article in proceedings

SN - 978-1-4673-0724-6

SP - 57

EP - 62

BT - 2012 Fourth International Workshop on Quality of Multimedia Experience (QoMEX)

PB - IEEE

T2 - 4th International Workshop onQuality of Multimedia Experience (QoMEX)

Y2 - 5 July 2012 through 7 July 2012

ER -

How to evaluate objective video quality metrics reliably

Abstract

Conference

Keywords

Access to Document

OpenUrl availability

Fingerprint

Cite this