PhD defense of Zhe ZHENG

Zhe ZHENG

Towards stable and reliable neural networks for multi-image restoration problems

Abstract

The restoration of images and videos from degraded multi-frame observations is a fundamental challenge in computer vision. While deep learning has significantly advanced this field, transitioning these models to practical, safety-critical deployments reveals critical limitations regarding efficiency, stability, and reliability. This thesis investigates and resolves these challenges from two related multi-image restoration domains: supervised video denoising and self-supervised burst satellite image super-resolution.

First, we address the efficiency and temporal consistency of video processing models in low-latency settings. We demonstrate that while recent Multi-Input Multi-Output (MIMO) feedforward architectures are computationally efficient, they suffer from severe quality decay at boundaries of output stacks (output window) of frames and step-wise motion artifacts at the transitions of output stacks. To resolve this, we introduce two strategies: Recurrence Across Stacks (RAS) and Output Stack Overlap (OSO). They effectively eliminate temporal discontinuities and improve overall output quality by expanding the temporal receptive field.

Second, we investigate the temporal aggregation capabilities of recurrent networks. Through empirical analysis on static sequences, we identify that despite of infinite temporal receptive field in theory, the recurrent networks trained via standard Backpropagation Through Time (BPTT) on short sequences fail to match optimal performance by denoising the running average due to a substantial training-inference distribution shift. Inspired by optimal recursive estimators, we study the incorporation and propagation of explicit uncertainty estimates along with the recurrent state, to help the network dynamically weigh past information against new observations. At the same time, we propose a Noise-Level Augmentation (NLA) training strategy, which successfully forces the network to aggregate information aggressively and significantly boosts peak performance. However, it exposes a critical performance-stability trade-off. Specifically, this aggressive temporal aggregation triggers severe dynamical instability and recursive error accumulation, leading to divergence over long sequences. To overcome this divergence, we propose a novel, input-dependent Jacobian-based regularization method that stabilizes the recurrent network while preserving its enhanced expressivity.

Finally, we tackle the challenge of reliability in safety-critical domains, where providing a single deterministic point estimate is insufficient, due to the ill-posed nature of the problem. Reflecting the reliability of the results requires quantifying the inherent uncertainty. Standard uncertainty estimation methods require ground-truth data, which is often impossible to acquire in practical applications like burst satellite image super-resolution. To overcome this fundamental bottleneck, we propose a novel loss function that extends the Gaussian Negative Log-Likelihood to self-supervised settings. This approach enables the accurate quantification of aleatoric uncertainty using only degraded observations, matching the optimal estimators obtained under supervised learning and allowing for reliable, uncertainty-aware reconstruction.

PhD supervisors

• Pablo ARIAS, Assistant Professor, Universitat Pompeu Fabra
• Gabriele FACCIOLO,  Professor, Centre Borelli, ENS Paris-Saclay

Defense committee

• Mme Coloma BALLESTER, Full professor, Universitat Pompeu Fabra  (Reviewer)
• M. Matias VALDENEGRO-TORO, Assistant professor, University of Groningen  (Reviewer)
• M. Andrés ALMANSA, Directeur de recherche, MAP5, Université Paris Cité  (Examiner)
• M. Thomas TANAY, Senior research scientist, Huawei Technologies Co., Ltd., Noah’s Ark Lab (Examiner)

Publications