File Download

There are no files associated with this item.

  Links for fulltext
     (May Require Subscription)
Supplementary

Article: Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation

TitleNeural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation
Authors
KeywordsVideo-based Characters
Deep Learning
Neural Rendering
Learning Dynamic Texture
Rendering-to-Video Translation
Issue Date2020
PublisherInstitute of Electrical and Electronics Engineers. The Journal's web site is located at https://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=2945
Citation
IEEE Transactions on Visualization and Computer Graphics, 2020, Epub 2020-05-26 How to Cite?
AbstractSynthesizing realistic videos of humans using neural networks has been a popular alternative to the conventional graphics-based rendering pipeline due to its high efficiency. Existing works typically formulate this as an image-to-image translation problem in 2D screen space, which leads to artifacts such as over-smoothing, missing body parts, and temporal instability of fine-scale detail, such as pose-dependent wrinkles in the clothing. In this paper, we propose a novel human video synthesis method that approaches these limiting factors by explicitly disentangling the learning of time-coherent fine-scale details from the embedding of the human in 2D screen space. More specifically, our method relies on the combination of two convolutional neural networks (CNNs). Given the pose information, the first CNN predicts a dynamic texture map that contains time-coherent high-frequency details, and the second CNN conditions the generation of the final video on the temporally coherent output of the first CNN. We demonstrate several applications of our approach, such as human reenactment and novel view synthesis from monocular video, where we show significant improvement over the state of the art both qualitatively and quantitatively.
Persistent Identifierhttp://hdl.handle.net/10722/293922
ISSN
2023 Impact Factor: 4.7
2023 SCImago Journal Rankings: 2.056
ISI Accession Number ID

 

DC FieldValueLanguage
dc.contributor.authorLIU, L-
dc.contributor.authorXu, W-
dc.contributor.authorHabermann, M-
dc.contributor.authorZollhoefer, M-
dc.contributor.authorBernard, F-
dc.contributor.authorKim, H-
dc.contributor.authorWang, WP-
dc.contributor.authorTheobalt, C-
dc.date.accessioned2020-11-23T08:23:47Z-
dc.date.available2020-11-23T08:23:47Z-
dc.date.issued2020-
dc.identifier.citationIEEE Transactions on Visualization and Computer Graphics, 2020, Epub 2020-05-26-
dc.identifier.issn1077-2626-
dc.identifier.urihttp://hdl.handle.net/10722/293922-
dc.description.abstractSynthesizing realistic videos of humans using neural networks has been a popular alternative to the conventional graphics-based rendering pipeline due to its high efficiency. Existing works typically formulate this as an image-to-image translation problem in 2D screen space, which leads to artifacts such as over-smoothing, missing body parts, and temporal instability of fine-scale detail, such as pose-dependent wrinkles in the clothing. In this paper, we propose a novel human video synthesis method that approaches these limiting factors by explicitly disentangling the learning of time-coherent fine-scale details from the embedding of the human in 2D screen space. More specifically, our method relies on the combination of two convolutional neural networks (CNNs). Given the pose information, the first CNN predicts a dynamic texture map that contains time-coherent high-frequency details, and the second CNN conditions the generation of the final video on the temporally coherent output of the first CNN. We demonstrate several applications of our approach, such as human reenactment and novel view synthesis from monocular video, where we show significant improvement over the state of the art both qualitatively and quantitatively.-
dc.languageeng-
dc.publisherInstitute of Electrical and Electronics Engineers. The Journal's web site is located at https://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=2945-
dc.relation.ispartofIEEE Transactions on Visualization and Computer Graphics-
dc.rightsIEEE Transactions on Visualization and Computer Graphics. Copyright © Institute of Electrical and Electronics Engineers.-
dc.rights©20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.-
dc.subjectVideo-based Characters-
dc.subjectDeep Learning-
dc.subjectNeural Rendering-
dc.subjectLearning Dynamic Texture-
dc.subjectRendering-to-Video Translation-
dc.titleNeural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation-
dc.typeArticle-
dc.identifier.emailWang, WP: wenping@cs.hku.hk-
dc.identifier.authorityWang, WP=rp00186-
dc.description.naturelink_to_subscribed_fulltext-
dc.identifier.doi10.1109/TVCG.2020.2996594-
dc.identifier.pmid32746256-
dc.identifier.scopuseid_2-s2.0-85114303493-
dc.identifier.hkuros318953-
dc.identifier.volumeEpub 2020-05-26-
dc.identifier.spage1-
dc.identifier.epage1-
dc.identifier.isiWOS:000692890200013-
dc.publisher.placeUnited States-

Export via OAI-PMH Interface in XML Formats


OR


Export to Other Non-XML Formats