An in-depth study of the deep learning pipeline

Despite the tremendous success of neural networks in many applications, progress in understanding their performance and limits is slow. While universal approximation theorems bring the promise of machines capable of approximating any complex mapping, these results are generic asymptotic statements that provide little insight into practical applications. Indeed, the complexity of any inquiry arises from the fact that three elements are necessaryto train a neural network: a dataset, a neural architecture, and an optimizer. Mostavailable theory focuses on only one of these elements at a time, and little is knownabout their interactions. This thesis develops an analysis of each of these elements andtheir interactions, theoretically and practically, through the lens of a particularly puzzling application, monocular depth estimation. We start by analyzing the datasets, which play the role of the shadows projected onto thewall of Plato's cave, whose inhabitants are neural networks. We study how their definition impacts neural networks and how the design of synthetic datasets can be leveraged to better understand the performance and limits of each neural architecture.Further analysis of the properties of neural architectures leads us to discuss means to adapt them to a new task or dataset, using two fundamental applications as testbeds:monocular depth estimation and video denoising methods. By giving counterexamples, we also point out how the universal approximation theorems do not apply to the practical setting.Finally, we dive into the theory of optimizers and describe how they define the structure of learning machines. We focus on the concept of stability of the optimization process and propose the use of explicit bounds on a network's parameters and updates, ensuring tha tits output will be bounded regardless of its width and depth. We finally derive a new form for classical neural optimizers that favor their stability.