Ref-NPR: Reference-Based Non-Photorealistic Radiance Fields for Controllable Scene Stylization

CVPR 2023

Yuechen Zhang1,2, Zexin He1, Jinbo Xing1, Xufeng Yao1, Jiaya Jia1,2
1The Chinese University of Hong Kong, 2SmartMore
Paper arXiv Video Code Data
* Given a 2D stylized view, Ref-NPR can stylize the whole nerf representation.
In following examples, style reference in each case is generated by hand drawing, 2D style transfer, or text-driven controllable generation (ControlNet).

Abstract

Existing 3D scene stylization methods employ an arbitrary style reference to transfer textures and colors as styles without establishing meaningful semantic correspondences. We present Reference-Based Non-Photorealistic Radiance Fields, i.e., Ref-NPR. It is a controllable scene stylization method utilizing radiance fields to stylize a 3D scene, with a single stylized 2D view taken as reference. To achieve decent results, we propose a ray registration process based on the stylized reference view to obtain pseudo-ray supervision in novel views, and exploit the semantic correspondence in content images to fill occluded regions with perceptually similar styles. Combining these operations, Ref-NPR generates non-photorealistic and continuous novel view sequences with a single reference while obtaining reasonable stylization in occluded regions. Experiments show that Ref-NPR significantly outperforms other scene and video stylization methods in terms of both visual quality and semantic correspondence.


Ref-NPR propagates 2D style reference to 3D radiance fields with Reference-Ray Registeration and Templated-Based Semantic Correspondence, thus obtaining better visual quality and visual correspondence.

Video

Comparisons

We compare Ref-NPR with two recent scene stylization methods, ARF and SNeRF, and also include a reference-based video stylization method by Texler for a more comprehensive comparison.

Content

Reference

Texler

ARF

SNerf

Ref-NPR

Extensions

Adaptation to 2D Stylization

Although Ref-NPR itself requires a stylized reference view, it can be extended to use arbitrary style images as reference. We use three different 2D stylization methods to generate three reference views of the same scene, each with a slightly different stylization, and then feed them into Ref-NPR to enable rendering of stylized novel views. This extension enables Ref-NPR to work with arbitrary style reference images.

Original Content

Arbitrary Style

2D Reference by NNST

2D Reference by Gatys

2D Reference by AdaIN

Appearance Editing

Added ketchup in the reference image is demonstrated properly in the stylized scene.

Controllability

This extension allows great flexibility and controllability regarding to the provided 2D reference.

Multi-Reference

Ref-NPR can be easily accommodated to multi-reference scenarios, by registering rays using all stylized reference views in R3, and by expanding the capacity of styles and content features in TCM. With two additional stylized views, better feature matching is obtained from richer style contents.

Scene with a single reference

Scene with multiple references

BibTeX

@InProceedings{Zhang_2023_CVPR,
        author    = {Zhang, Yuechen and He, Zexin and Xing, Jinbo and Yao, Xufeng and Jia, Jiaya},
        title     = {Ref-NPR: Reference-Based Non-Photorealistic Radiance Fields for Controllable Scene Stylization},
        booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
        month     = {June},
        year      = {2023},
        pages     = {4242-4251}
    }