Video Frame Interpolation via Cyclic Fine-Tuning and Asymmetric Reverse Flow

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

77 Downloads (Pure)

Abstract

The objective in video frame interpolation is to predict additional in-between frames in a video while retaining natural motion and good visual quality. In this work, we use a convolutional neural network (CNN) that takes two frames as input and predicts two optical flows with pixelwise weights. The flows are from an unknown in-between frame to the input frames. The input frames are warped with the predicted flows, multiplied by the predicted weights, and added to form the in-between frame. We also propose a new strategy to improve the performance of video frame interpolation models: we reconstruct the original frames using the learned model by reusing the predicted frames as input for the model. This is used during inference to fine-tune the model so that it predicts the best possible frames. Our model outperforms the publicly available state-of-the-art methods on multiple datasets.
Original languageEnglish
Title of host publicationProceedings of 2019 Scandinavian Conference on Image Analysis
PublisherSpringer
Publication date2019
Pages311-323
ISBN (Print)9783030202057
DOIs
Publication statusPublished - 2019
EventScandinavian Conference on Image Analysis - Norrköpings Visualisering Center, Norrköping, Sweden
Duration: 11 Jun 201913 Jun 2019
http://ssba.org.se/scia2019/

Conference

ConferenceScandinavian Conference on Image Analysis
LocationNorrköpings Visualisering Center
CountrySweden
CityNorrköping
Period11/06/201913/06/2019
Internet address
SeriesLecture Notes in Computer Science
Volume11482
ISSN0302-9743

Keywords

  • Computer Science
  • Image Processing and Computer Vision
  • Pattern Recognition
  • Computer Graphics
  • Artificial Intelligence
  • Computer Communication Networks
  • Slow motion
  • Video frame interpolation
  • Convolutional neural networks

Cite this

Hannemose, M., Jensen, J. N., Einarsson, G., Wilm, J., Dahl, A. B., & Frisvad, J. R. (2019). Video Frame Interpolation via Cyclic Fine-Tuning and Asymmetric Reverse Flow. In Proceedings of 2019 Scandinavian Conference on Image Analysis (pp. 311-323). Springer. Lecture Notes in Computer Science, Vol.. 11482 https://doi.org/10.1007/978-3-030-20205-7_26
Hannemose, Morten ; Jensen, Janus Nørtoft ; Einarsson, Gudmundur ; Wilm, Jakob ; Dahl, Anders Bjorholm ; Frisvad, Jeppe Revall. / Video Frame Interpolation via Cyclic Fine-Tuning and Asymmetric Reverse Flow. Proceedings of 2019 Scandinavian Conference on Image Analysis. Springer, 2019. pp. 311-323 (Lecture Notes in Computer Science, Vol. 11482).
@inproceedings{3a0dea9b08824cffa9b190236de937f2,
title = "Video Frame Interpolation via Cyclic Fine-Tuning and Asymmetric Reverse Flow",
abstract = "The objective in video frame interpolation is to predict additional in-between frames in a video while retaining natural motion and good visual quality. In this work, we use a convolutional neural network (CNN) that takes two frames as input and predicts two optical flows with pixelwise weights. The flows are from an unknown in-between frame to the input frames. The input frames are warped with the predicted flows, multiplied by the predicted weights, and added to form the in-between frame. We also propose a new strategy to improve the performance of video frame interpolation models: we reconstruct the original frames using the learned model by reusing the predicted frames as input for the model. This is used during inference to fine-tune the model so that it predicts the best possible frames. Our model outperforms the publicly available state-of-the-art methods on multiple datasets.",
keywords = "Computer Science, Image Processing and Computer Vision, Pattern Recognition, Computer Graphics, Artificial Intelligence, Computer Communication Networks, Slow motion, Video frame interpolation, Convolutional neural networks",
author = "Morten Hannemose and Jensen, {Janus N{\o}rtoft} and Gudmundur Einarsson and Jakob Wilm and Dahl, {Anders Bjorholm} and Frisvad, {Jeppe Revall}",
year = "2019",
doi = "10.1007/978-3-030-20205-7_26",
language = "English",
isbn = "9783030202057",
pages = "311--323",
booktitle = "Proceedings of 2019 Scandinavian Conference on Image Analysis",
publisher = "Springer",

}

Hannemose, M, Jensen, JN, Einarsson, G, Wilm, J, Dahl, AB & Frisvad, JR 2019, Video Frame Interpolation via Cyclic Fine-Tuning and Asymmetric Reverse Flow. in Proceedings of 2019 Scandinavian Conference on Image Analysis. Springer, Lecture Notes in Computer Science, vol. 11482, pp. 311-323, Scandinavian Conference on Image Analysis, Norrköping, Sweden, 11/06/2019. https://doi.org/10.1007/978-3-030-20205-7_26

Video Frame Interpolation via Cyclic Fine-Tuning and Asymmetric Reverse Flow. / Hannemose, Morten; Jensen, Janus Nørtoft; Einarsson, Gudmundur; Wilm, Jakob; Dahl, Anders Bjorholm; Frisvad, Jeppe Revall.

Proceedings of 2019 Scandinavian Conference on Image Analysis. Springer, 2019. p. 311-323 (Lecture Notes in Computer Science, Vol. 11482).

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

TY - GEN

T1 - Video Frame Interpolation via Cyclic Fine-Tuning and Asymmetric Reverse Flow

AU - Hannemose, Morten

AU - Jensen, Janus Nørtoft

AU - Einarsson, Gudmundur

AU - Wilm, Jakob

AU - Dahl, Anders Bjorholm

AU - Frisvad, Jeppe Revall

PY - 2019

Y1 - 2019

N2 - The objective in video frame interpolation is to predict additional in-between frames in a video while retaining natural motion and good visual quality. In this work, we use a convolutional neural network (CNN) that takes two frames as input and predicts two optical flows with pixelwise weights. The flows are from an unknown in-between frame to the input frames. The input frames are warped with the predicted flows, multiplied by the predicted weights, and added to form the in-between frame. We also propose a new strategy to improve the performance of video frame interpolation models: we reconstruct the original frames using the learned model by reusing the predicted frames as input for the model. This is used during inference to fine-tune the model so that it predicts the best possible frames. Our model outperforms the publicly available state-of-the-art methods on multiple datasets.

AB - The objective in video frame interpolation is to predict additional in-between frames in a video while retaining natural motion and good visual quality. In this work, we use a convolutional neural network (CNN) that takes two frames as input and predicts two optical flows with pixelwise weights. The flows are from an unknown in-between frame to the input frames. The input frames are warped with the predicted flows, multiplied by the predicted weights, and added to form the in-between frame. We also propose a new strategy to improve the performance of video frame interpolation models: we reconstruct the original frames using the learned model by reusing the predicted frames as input for the model. This is used during inference to fine-tune the model so that it predicts the best possible frames. Our model outperforms the publicly available state-of-the-art methods on multiple datasets.

KW - Computer Science

KW - Image Processing and Computer Vision

KW - Pattern Recognition

KW - Computer Graphics

KW - Artificial Intelligence

KW - Computer Communication Networks

KW - Slow motion

KW - Video frame interpolation

KW - Convolutional neural networks

U2 - 10.1007/978-3-030-20205-7_26

DO - 10.1007/978-3-030-20205-7_26

M3 - Article in proceedings

SN - 9783030202057

SP - 311

EP - 323

BT - Proceedings of 2019 Scandinavian Conference on Image Analysis

PB - Springer

ER -

Hannemose M, Jensen JN, Einarsson G, Wilm J, Dahl AB, Frisvad JR. Video Frame Interpolation via Cyclic Fine-Tuning and Asymmetric Reverse Flow. In Proceedings of 2019 Scandinavian Conference on Image Analysis. Springer. 2019. p. 311-323. (Lecture Notes in Computer Science, Vol. 11482). https://doi.org/10.1007/978-3-030-20205-7_26