Physics-aware deep learning aims to explore the potential of AI methods to advance scientific research in

modeling complex natural phenomena. One area where this idea holds great promise is in modeling the

complex dynamics of natural phenomena, such as in climate science. Among recent developments, the

concept of developing foundation models that learn from multiple physical domains is emerging as one

of the fundamental challenges in this field.

Foundation models have become prominent in domains like natural language processing and computer

vision. Trained with large quantities of data using self-supervision, they can be used or adapted for

downstream tasks while benefiting from pre-training on vast amounts of data. Replicating this framework

in scientific domains is currently under investigation, with recent successes in areas such as weather

forecasting (Pathak 2022, Nguyen 2023, Kochkov 2024, Bodnar 2024) and modeling diverse dynamical

phenomena (Hao 2024). Within this context, the objective of the internship is to investigate the potential

of this approach in two directions, focusing on the modeling of ocean dynamics:

– Leveraging the complementarity of multiple information sources for modeling complex

dynamical phenomena. Ocean dynamics, for example, is a complex spatio-temporal process that

is only partially observed through satellites or in situ sensors. These sensors measure different

quantities such as Sea Surface Height Anomaly (SLA), Sea Surface Temperature (SST), and Sea

Surface Salinity (SSS). All are major variables of ocean equations and offer partial but

complementary information about ocean dynamics. Exploiting this complementarity is

challenging due to the heterogeneity of the data, differences in spatial and temporal resolution,

and incompleteness. We will leverage from recent advances in vision transformers and the

modeling of dynamical processes to address this problem (Bodnar 2024).

– Addressing the generalization ability of data-driven models for climate applications. Statistical

models generalize well only if the test distribution is identical to the training distribution. Given

the complexity and variety of contexts, this is rarely the case for dynamical phenomena observed

in nature. Data-driven models must adapt quickly and with minimal observations to new

situations. The internship will explore this problem through the lens of in-context learning (Chen

2024), an adaptation scheme compatible with the deployment of foundation models.

Bodnar C, Bruinsma WP, Lucic A, Brandstetter J, Garvan P et al. AURORA: A FOUNDATION MODEL OF THE ATMOSPHERE. In: ; 2024.

https://arxiv.org/abs/2405.13063

Chen, W., Song, J., Ren, P., Subramanian, S., Morozov, D., & Mahoney, M. W. (2024). Data-Efficient Operator Learning via

Unsupervised Pretraining and In-Context Learning. 1–21. http://arxiv.org/abs/2402.15734

Hao, Z., Su, C., Liu, S., Berner, J., Ying, C., Su, H., Anandkumar, A., Song, J., & Zhu, J. (2024). DPOT: Auto-Regressive Denoising

Operator Transformer for Large-Scale PDE Pre-Training. Icml. http://arxiv.org/abs/2403.03542

Kirchmeyer, M., Yin, Y., Donà, J., Baskiotis, N., Rakotomamonjy, A., & Gallinari, P. (2022). Generalizing to New Physical Systems via

Context-Informed Dynamics Model. ICML.

Kochkov D, Yuval J, Langmore I, et al. Neural General Circulation Models. In: ArXiv:2311.07222v2. ; 2024.

Nguyen, T., Brandstetter, J., Kapoor, A., Gupta, J. K., and Grover, A. Climax: A foundation model for weather and climate.

arXiv:2301.10343, 2023.

Pathak, J., Subramanian, S., Harrington, P., Raja, S., Chattopadhyay, A., Mardani, M., Kurth, T., Hall, D., Li, Z., Azizzadenesheli, K., et

al. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators.