그림 1. 기존 얼굴 변형 애플리케이션과 제안하는 시스템을 통한 변형 예시: (1) 원본 영상, (2) [1]을 이용한 의한 변형 결과, (3) [2]를 이용한 변형 결과, (4) 제안하는 시스템을 이용한 변형 결과 Fig. 1. Examples of face manipulation via the existing applications and the proposed system: (1) original image, (2) manipulation result using [1], (3) manipulation result using [2], (4) manipulation result using the proposed system
그림 2. 제안하는 얼굴 변형 시스템의 흐름 Fig. 2. Flow of the proposed face manipulation system
그림 3. 윤곽 랜드마크에 대한 대응점 탐색 결과: (좌) 영상에서 검출된 랜드마크, (중앙) [18]의 방법에 따라 탐색된 대응점, (우) 제안된 시스템에서 수정된 방법에 따라 탐색된 대응점 Fig. 3. Correspondence searching results for contour landmarks: (left) landmarks detected from image, (middle) correspondences detected using the method in [18], (right) correspondences detected using the modified method in the proposed system
그림 4. 사용자 제약조건(제어 정점: 보라색 점, 변형될 정점: 녹색 점)의 설정과 메쉬 변형의 예시: (좌) 원본 메쉬 모델, (중앙) 사용자 제약조건의 설정, (우) 제어 정점의 이동에 따라 변형된 메쉬 Fig. 4. An example of setting user constraints (control vertices: pupple points, roi vertices: green points) and mesh deformation: (Left) original mesh model, (middle) setting user constraints, (right) deformed mesh according to the movement of the control vertices
그림 5. 제안된 시스템의 다중쓰레드 Fig. 5. Multithreading in the proposed system
그림 6. 비디오 프레임 대한 얼굴 변형 결과: (1) 원본 프레임, (2)~(4) 눈 주변, 턱, 코를 변형한 결과, (5) (2)~(4)의 변형을 모두 적용한 결과 Fig. 6. Results of face manipulation for video frames: (1) original frames, (2)~(4) results of manipulating the area around eyes, jaw, and nose, (5) results of applying all the manipulations from (2) to (4)
그림 7. 기존 애플리케이션과의 얼굴 변형 비교: (1) 원본 영상, (2) [1]으로 변형한 결과, (3) 제안된 시스템으로 변형한 결과 Fig. 7. Comparison of face manipulation results from the existing application : (1) original images, (2) results from [1], (3) results from the proposed system
그림 8. 기존 애플리케이션과의 얼굴 변형 비교: (1) [2]로 변형한 결과, (2) 제안된 시스템으로 변형한 결과 Fig. 8. Comparison of face manipulation results from the existing application : (1) results from [2], (2) results from the proposed system
표 1. 고속화 여부에 따른 얼굴 모델 피팅에 소요되는 시간 Table 1. The processing time for face model fitting w/o and w/ acceleration
표 2. 처리 단계별 소요 시간 Table 2. Processing time for each process
참고문헌
- Virtual Plastic Surgery Simulator, https://www.plastic-surgery-simulator.com (accessed July 30, 2018)
- Snow.me, https://snow.me (accessed July 30, 2018)
- H.-V. Chung and I.-K. Lee, "Image-Based Deformation of Objects in R eal Scenes," Proceedings of International Symposium on Visual Computing, pp.159-166, 2005, https://doi.org/10.1007/11595755_ 20.
- J. Park, B.-K. Seo and J.-I. Park, "[Poster] Interactive Deformation of Real Objects," Proceedings of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp.295-296, 2014, https://doi.org/10.1109/ISMAR.2014.6948457.
- Y. Zheng, X. Chen, M.-M. Cheng, K. Zhou, S.-M. Hu, and N. J. Mitra, "Interactive Images: Cuboid Proxies for Smart Image Manipulation," ACM Transactions on Graphics, Vol.31, No.4, pp.99:1-99:11, Jul. 2012, https://doi.org/10.1145/2185520.2185595.
- T. Chen, Z. Zhu, A. Shamir, S.-M. Hu, and D. Cohen-Or, "3-Sweep: E xtracting Editable Objects from a Single Photo," ACM Transactions on Graphics, Vol.32, No.6, pp.195:1-195:10, Nov. 2013, https://doi.org/10.1145/2508363.2508378.
- N. Kholgade, T. Simon, A. Efros, and Y. Sheikh, "3D Object Manipula tion in a Single Photograph Using Stock 3D Models," ACM Transactions on Graphics, Vol.33, No.4, pp.127:1-127:12, Jul. 2014, https://doi.org/10.1145/2601097.2601209.
- N. Haouchine, A. Petit, F. Roy, and S. Cotin, "[Poster] Deformed Reality: Proof of Concept and Preliminary Results," Proceedings of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp.166-167, 2017, https://doi.org/10.1109/ISMAR-Adjunct.2017.56.
- J. Thies, M. Zollhofer, M. Stamminger, C. Theobalt, and M. Niessner, "Face2Face: Real-Time Face Capture and Reenactment of RGB Videos," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2387-2395, 2016, https://doi.org/10.1109/CVPR.2016.262.
- K. Dale, K. Sunkavalli, M. K. Johnson, D. Vlasic, W. Matusik, and H. Pfister, "Video Face Replacement," ACM Transactions on Graphics, Vol.30, No.6, pp.130:1-130:10, Dec. 2011, https://doi.org/10.1145/2070781.2024164.
- C. Cao, Y. Weng, S. Zhou, Y. Tong, and K. Zhou, "FaceWarehouse: A 3D Facial Expression Database for Visual Computing," IEEE Transactions on Visualization and Computer Graphics, Vol.20, No.3, pp.413-425, Mar. 2014, https://doi.org/10.1109/TVCG.2013.249.
- C. Cao, H. Wu, Y. Weng, T. Shao, and K. Zhou, "Real-Time Facial Animation with Image-Based Dynamic Avatars," ACM Transactions on Graphics, Vol.35, No.4, pp.126:1-126:12, Jul. 2016, https://doi.org/10.1145/2897824.2925873.
- K. Nagano, J. Seo, J. Xing, L.Wei, Z. Li, S. Saito, A. Agarwal, J. Fursund, and H. Li, "paGAN: Real-Time Avatars Using Dynamic Textures," ACM Transactions on Graphics, Vol.37, No.6, pp.258:1-258:12, Dec. 2018, https://doi.org/10.1145/3272127.3275075.
- J. Park and J.-I. Park, "A Framework for Virtual 3D Manipulation of Fa ce in Video," Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp.649-650, 2018, https://doi.org/10.1109/VR.2018.8446445.
- S. Ren, X. Cao, Y. Wei, and J. Sun, "Face Alignment at 3000 FPS via Regressing Local Binary Features," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.1685-1692, 2014, https://doi.org/10.1109/CVPR.2014.218.
- V. Kazemi and J. Sullivan, "One Millisecond Face Alignment with an Ensemble of Regression Trees," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.1867-1874, 2014, https://doi.org/10.1109/CVPR.2014.241.
- M. Kowalski, J. Naruniec, and T. Trzcinski, "Deep Alignment Networ k: A Convolutional Neural Network for Robust Face Alignment," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2017.
- P. Huber, G. Hu, R. Tena, P. Mortazavian, P. Koppen, W. J. Christmas, M. Ratsch, and J. Kittler, "A Multiresolution 3D Morphable Face Mod el and Fitting Framework," Proceedings of the 11th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, 2016.
- O. Sorkine and M. Alexa, "As-Rigid-As-Possible Surface Modeling," Proceedings of the 5th Eurographics Symposium on Geometry Processing, pp.109-116, 2007.
- A. Telea, "An image inpainting technique based on the fast marching method," Journal of Graphics Tools, Vol.9, No.1, pp.23-34, 2004, https://doi.org/10.1080/10867651.2004.10487596.