Continual learning in dynamic environments is a challenge for large-scale machine learning models. This research addresses Domain Incremental Learning (DIL), a setting where the goal is to incrementally increase the input data scope of a model. More specifically, we investigate the possibility of using foundation models (FMs) as a fixed feature extractor combined with a PPCA that can be sequentially and accurately updated. Focusing on the classification of VHR remote sensing (RS) images, we show on the FLAIR#1 dataset that this simple DIL strategy achieves competitive accuracy compared to memory-based baselines across different pre-trained sources. We also compare different types of foundation models and highlight the importance of data diversity over data specialization to improve the quality of FMs.