Recently, Transformers have been shown to enhance the performance of multi-view stereo by enabling long-range feature interaction. In this work, we propose Window-based Transformers (WT) for local feature matching and global feature aggregation in multi-view stereo. We introduce a Window-based Epipolar Transformer (WET) which reduces matching redundancy by using epipolar constraints. Since point-to-line matching is sensitive to erroneous camera pose and calibration, we match windows near the epipolar lines. A second Shifted WT is employed for aggregating global information within cost volume. We present a novel Cost Transformer (CT) to replace 3D convolutions for cost volume regularization. In order to better constrain the estimated depth maps from multiple views, we further design a novel geometric consistency loss (Geo Loss) which punishes unreliable areas where multi-view consistency is not satisfied. Our WT multi-view stereo method (WT-MVSNet) achieves state-of-the-art performance across multiple datasets and ranks 1st on Tanks and Temples benchmark.
|Title of host publication
|Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022
|S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, A. Oh
|Neural information processing systems foundation
|Published - 2022
|36th Conference on Neural Information Processing Systems, NeurIPS 2022 - New Orleans, United States
Duration: 28 Nov 2022 → 9 Dec 2022
|Advances in Neural Information Processing Systems
|36th Conference on Neural Information Processing Systems, NeurIPS 2022
|28/11/22 → 9/12/22
Bibliographical notePublisher Copyright:
© 2022 Neural information processing systems foundation. All rights reserved.