Error Correction through Multimodal Interpretable Meta Conditions: Simulated Aerial Imagery Dataset

Noel Ngu*, Aditya Taparia*, Gerardo I. Simari*, Mario Leiva*, Ransalu Senanayake*, Paulo Shakarian*,
Arizona State University
*Indicates Equal Contribution
Download Dataset


Arizona State University is releasing a new aerial imagery dataset aimed at advancing object detection and creating new solutions for handling diverse data distributions in aerial imagery. The dataset is generated using the AirSim simulator and contains images taken under different weather conditions. Object-detection models fine-tuned on each dataset distribution are also provided as well to facilitate research in this area.

Introduction

This dataset contains images from the AirSim simulator in the City environment. The images are taken at random positions within the environment under various weather conditions such as fog, rain, snow, dust, and maple leaves. The dataset is aimed at advancing object detection by addressing diverse data distributions.
The idea of this dataset is that models trained on different distributions of data (which in this case is different distributions of weather) can be combined using an ensemble approach to effectively handle a separate distribution to enable better generalization and performance on diverse datasets.

Training Sets

It contains images from the AirSim simulator in the CityEnviron environment. The images are taken at random positions within the environment under weather conditions such as fog, rain, snow, dust, and maple leaves.

The GitHub repository contains datasets with different distributions:
  • dust: This dataset contains images where dust is the prominent weather condition.
  • fog: This dataset contains images where fog is the prominent weather condition.
  • maple_leaf: This dataset contains images where maple leaves are the prominent weather condition.
  • rain: This dataset contains images where rain is the prominent weather condition.
  • snow: This dataset contains images where the parameter for snow is set to a high value.
The bar charts below display the average intensity of each weather parameter in each training set. MY ALT TEXT

Test Set

The test set contains aerial imagery captured under mixed weather conditions using AirSim's drone vehicle. While the training sets each contain images with a particular prominent weather condition, the test set contains a variety of mixed weather conditions to emulate real-life scenarios and to test the generalization of models trained on the training sets.

The bar chart below display the average intensity of each weather parameter in the test set.

MY ALT TEXT

Implementation

The dataset contains various images captured at random positions within the City environment in Airsim. The following is a table containing the various parameters that were set in the AirSim simulator to produce the datasets with different distributions.
MY ALT TEXT

Data format

Each dataset adheres to the COCO format and includes three key folders: annotations, train, and val.

  • annotations: This folder contains the annotations for the train and validation datasets in the COCO format named custom_train.json and custom_val.json respectively.
  • train: This folder contains the images for the training set.
  • val: This folder contains the images for the validation dataset.

Models

Various models were trained on each distribution using Facebook's DeTR model. The models were trained with consistent hyperparameters, which include:

  • Number of epochs: 500
  • Learning rate: 0.00005
  • Weight decay: 0.0001
  • Max gradient norm: 0.01

This setup ensures that each model is optimized for its specific distribution while following a standardized training process.

The model weights for models trained on each training set are available for general use.

MY ALT TEXT
Each model has different strengths, one can see that the model on the left trained on maple leaves is able to distinguish between maple leaves and pedestrians whereas the one on the left trained on snow has some trouble.

BibTeX


        @misc{shakarian2025error,
          author = {Paulo Shakarian and Ransalu Senanayake and Gerardo I. Simari and Mario Leiva and Aditya Taparia and Noel Ngu},
          title = {Error Correction through Multimodal Interpretable Meta Conditions: Simulated Aerial Imagery Dataset},
          year = {2025},
          url = {https://neurosymbolic.asu.edu/metacognition/}
        }