Search and Rescue or surveillance applications from en embedded moving camera yield challenging computer vision problems as both very high precision/recall and real-time performance are required. However, in these contexts, it is often sufficient to assess the detection of each object of interest only once in the area spanned by the camera, with the idea that what matters is to be aware of its existence, its time to detection being a secondary objective. Taking advantage of this point, we describe a sparse box proposal controlled by tracking and designed to generate a small number of boxes per frame covering each object at least once in the video. Our sparse box proposal adapts to the budget allowed for single box classification and is able to achieve relevant results even on challenging situations while comfortably dealing with real time requirements.