AVSS2019 Challenge



Challenge

The increasing progress of transportation systems has led to a drastic increase in the demand for smart systems to monitor traffic and street safety. One important information for such systems is the locations and number of specific types of vehicles, and it is goal of this Challenge to provide a comprehensive performance evaluation to state-of-the-art detection and counting algorithms. The challenge is based on the UA-DETRAC dataset, a real-world multi-object detection and multi-object tracking benchmark. The dataset is composed of traffic video sequences annotated with vehicle bounding boxes and trajectories at various challenging levels. The UA-DETRAC challenge will be held in conjunction with the International Workshop on Traffic and Street Surveillance for Safety and Security (T4S), and works with performance over the baseline thresholds are encouraged to be submitted and peer reviewed for acceptance.

Participation

Participants of the Challenge must submit final results on the UA-DETRAC testing dataset with a document to briefly describe the applied methodology. All the works, whose results are above the set thresholds, will be with performance over the baseline thresholds are encouraged to submitted and peer reviewed for acceptance in the workshop proceeding (in association with the AVSS 2019 conference). Results and authorship of the papers that participated to the challenge and that scored more than the threshold will be summarized in one final paper in the main conference.

Challenge organization and evaluation

  • Dataset specifics: The UA-DETRAC dataset consists of 10 hours of videos captured with a Cannon EOS 550D camera at 24 different locations at Beijing and Tianjin in China. The videos are recorded at 25 frames per seconds (fps), with resolution of 960 × 540 pixels. There are more than 140 thousand frames in the UA-DETRAC dataset and 8,250 vehicles that are manually annotated, leading to a total of 1.21 million labeled bounding boxes of objects. The UA-DETRAC dataset is divided into training (UA-DETRAC-train) and testing (UA-DETRAC-test) sets, with 60 and 40 sequences, respectively. Training videos are taken at different locations from the testing videos, but similar traffic conditions and attributes are ensured.
  • Tasks: Following the UA-DETRAC organization, the challenge is divided in two main tasks, vehicle detection and vehicle counting. To that end, we improve our annotations with more vehicle type and color (see the V3 XML files). Specifically, the 6 main vehicle types are listed as follows:
    1. Sedan including Sedan and Police
    2. SUV including SUV and Hatchback
    3. Van including MiniVan and Van
    4. Taxi
    5. Bus
    6. Truck including Truck Pickup, Truck Box Med, Truck Box Large, Truck Flatbed, and Truck Util
    The 10 main vehicle colors, determined based on general car color popularity (from statistics from DMV and Wikipedia, the source of data are at https://en.wikipedia.org/wiki/Car_color_popularity), are listed as follows:
    1. White
    2. Silver
    3. Black
    4. Gray
    5. Blue
    6. Red
    7. Brown
    8. Green
    9. Yellow including Yellow, Orange, and Golden
    10. Multi
    Participators of the Challenge can choose to participate one or both of the tasks and contribute innovative solutions.
  • Vehicle Detection. The goal of the vehicle detection task is to locate different types of vehicles in the videos. Submissions to the vehicle detection task must include the bounding boxes and types of each vehicle from the pre-defined 6 main vehicle types in the testing video sequences. We use the mean average precision (mAP) score of the precision vs. recall (PR) curve to indicate the performance of vehicle detection methods, where the hit/miss threshold of the overlap between a detected bounding box and a ground truth bounding box set to 0.7.
  • Vehicle Counting. The goal of the vehicle counting task is to count the specific kind of vehicle in a video sequence, e.g., red sedan in the video. Submissions to the vehicle counting task must include the estimated numbers of vehicles from the pre-defined 6 vehicle types and 10 vehicle colors in the testing video sequences. Notably, for vehicles of multiple color (e.g. taxi and bus), we label their color using the "multi" type. We use the mean counting error (MCE) over all vehicle types and all vehicle colors in the testing sequences. Notably, the vehicles with the same ID in the video frames are regarded as one count.


    • Result and paper submission

    The result should be submitted to the official mailbox no more than 3 times. Specifically, participants must:

    1. Register an account at here and activate it by a verification email.
    2. Download the UA-DETRAC dataset.
    3. Send your detection/counting result files with the defined format to us by mail. Besides, a short document describing the applied methodology using the provided template is a must.
    4. Check your email for the evaluation results for the UA-DETRAC-test set after the challenge is closed.
    5. The paper related with the challenge must be submitted through a Content Management Toolkit. A template will be provided to the authors by considering the guidelines given by AVSS 2019 Workshop chairs. The corresponding author email indicated in the paper must be the same as the one used in step 1.