Add benchmarks

README.md

@@ -8,7 +8,6 @@ NVIDIA DeepStream SDK 6.2 / 6.1.1 / 6.1 / 6.0.1 / 6.0 configuration for YOLO mod
 
 ### Future updates
 
-* Models benchmarks
 * DeepStream tutorials
 * Dynamic batch-size
 * Updated INT8 calibration
@@ -19,6 +18,7 @@ NVIDIA DeepStream SDK 6.2 / 6.1.1 / 6.1 / 6.0.1 / 6.0 configuration for YOLO mod
 
 * Support for INT8 calibration
 * Support for non square models
+* Models benchmarks
 * **Support for Darknet YOLO models (YOLOv4, etc) using cfg and weights conversion with GPU post-processing**
 * **Support for YOLO-NAS, PPYOLOE+, PPYOLOE, YOLOX, YOLOR, YOLOv8, YOLOv7, YOLOv6 and YOLOv5 using ONNX conversion with GPU post-processing**
 
@@ -168,9 +168,36 @@ topk = 300
 
 **NOTE**: ** = The YOLOv4 is trained with the trainvalno5k set, so the mAP is high on val2017 test
 
-| DeepStream         | Precision | Resolution | IoU=0.5:0.95 | IoU=0.5 | IoU=0.75 | FPS<br />(without display) |
+**NOTE**: The p3.2xlarge instance (AWS) seems to max out at 625-635 FPS on DeepStream even using lighter models
-|:------------------:|:---------:|:----------:|:------------:|:-------:|:--------:|:--------------------------:|
+
-| Coming soon        | FP16      | 640        |         |    |     |                      |
+| DeepStream       | Precision | Resolution | IoU=0.5:0.95 | IoU=0.5 | IoU=0.75 | FPS<br />(without display) |
+|:----------------:|:---------:|:----------:|:------------:|:-------:|:--------:|:--------------------------:|
+| YOLO-NAS L       | FP16      | 640        | 0.484        | 0.658   | 0.532    | 235.27                     |
+| YOLO-NAS M       | FP16      | 640        | 0.480        | 0.651   | 0.524    | 287.39                     |
+| YOLO-NAS S       | FP16      | 640        | 0.442        | 0.614   | 0.485    | 478.52                     |
+| PP-YOLOE+_x      | FP16      | 640        | 0.        | 0.   | 0.    |                      |
+| PP-YOLOE+_l      | FP16      | 640        | 0.        | 0.   | 0.    |                      |
+| PP-YOLOE+_m      | FP16      | 640        | 0.        | 0.   | 0.    |                      |
+| PP-YOLOE+_s      | FP16      | 640        | 0.424        | 0.594   | 0.464    | 476.13                     |
+| PP-YOLOE-s (400) | FP16      | 640        | 0.423        | 0.589   | 0.463    | 461.23                     |
+| YOLOX-x          | FP16      | 640        | 0.447        | 0.616   | 0.483    | 125.40                     |
+| YOLOX-l          | FP16      | 640        | 0.430        | 0.598   | 0.466    | 193.10                     |
+| YOLOX-m          | FP16      | 640        | 0.397        | 0.566   | 0.431    | 298.61                     |
+| YOLOX-s          | FP16      | 640        | 0.335        | 0.502   | 0.365    | 522.05                     |
+| YOLOX-s legacy   | FP16      | 640        | 0.375        | 0.569   | 0.407    | 518.52                     |
+| YOLOX-Darknet    | FP16      | 640        | 0.414        | 0.595   | 0.453    | 212.88                     |
+| YOLOX-Tiny       | FP16      | 640        | 0.274        | 0.427   | 0.292    | 633.95                     |
+| YOLOX-Nano       | FP16      | 640        | 0.212        | 0.342   | 0.222    | 633.04                     |
+| YOLOv8x          | FP16      | 640        | 0.499        | 0.669   | 0.545    | 130.49                     |
+| YOLOv8l          | FP16      | 640        | 0.491        | 0.660   | 0.535    | 180.75                     |
+| YOLOv8m          | FP16      | 640        | 0.468        | 0.637   | 0.510    | 278.08                     |
+| YOLOv8s          | FP16      | 640        | 0.415        | 0.578   | 0.453    | 493.45                     |
+| YOLOv8n          | FP16      | 640        | 0.343        | 0.492   | 0.373    | 627.43                     |
+| YOLOv7           | FP16      | 640        | 0.        | 0.   | 0.    |                      |
+| YOLOv6s 3.0      | FP16      | 640        | 0.        | 0.   | 0.    |                      |
+| YOLOv5s 7.0      | FP16      | 640        | 0.        | 0.   | 0.    |                      |
+| YOLOv4           | FP16      | 640        | 0.        | 0.   | 0.    |                      |
+| YOLOv3           | FP16      | 640        | 0.        | 0.   | 0.    |                      |
 
 ##
 
@@ -926,6 +953,8 @@ model-file=yolov4.weights
 deepstream-app -c deepstream_app_config.txt
 ```
 
+**NOTE**: The TensorRT engine file may take a very long time to generate (sometimes more than 10 minutes).
+
 **NOTE**: If you want to use YOLOv2 or YOLOv2-Tiny models, change the `deepstream_app_config.txt` file before run it
 
 ```

config_infer_primary_yolox.txt

@@ -1,6 +1,6 @@
 [property]
 gpu-id=0
-net-scale-factor=0
+net-scale-factor=1
 model-color-format=0
 onnx-file=yolox_s.onnx
 model-engine-file=yolox_s.onnx_b1_gpu0_fp32.engine

docs/PPYOLOE.md

@@ -107,13 +107,19 @@ parse-bbox-func-name=NvDsInferParseYoloE
 
 **NOTE**: If you use the **legacy** model, you should edit the `config_infer_primary_ppyoloe.txt` file.
 
+**NOTE**: The **PP-YOLOE+ and PP-YOLOE legacy** do not resize the input with padding. To get better accuracy, use
+
+```
+maintain-aspect-ratio=0
+```
+
 **NOTE**: The **PP-YOLOE+** uses zero mean normalization on the image preprocess. It is important to change the `net-scale-factor` according to the trained values.
 
 ```
 net-scale-factor=0.0039215697906911373
 ```
 
-**NOTE**: The **PP-YOLOE (legacy)** uses normalization on the image preprocess. It is important to change the `net-scale-factor` and `offsets` according to the trained values.
+**NOTE**: The **PP-YOLOE legacy** uses normalization on the image preprocess. It is important to change the `net-scale-factor` and `offsets` according to the trained values.
 
 Default: `mean = 0.485, 0.456, 0.406` and `std = 0.229, 0.224, 0.225`
 
@@ -143,4 +149,6 @@ config-file=config_infer_primary_ppyoloe_plus.txt
 deepstream-app -c deepstream_app_config.txt
 ```
 
+**NOTE**: The TensorRT engine file may take a very long time to generate (sometimes more than 10 minutes).
+
 **NOTE**: For more information about custom models configuration (`batch-size`, `network-mode`, etc), please check the [`docs/customModels.md`](customModels.md) file.

docs/YOLONAS.md

@@ -1,4 +1,4 @@
-# YOLONAS usage
+# YOLO-NAS usage
 
 **NOTE**: The yaml file is not required.
 
@@ -149,6 +149,13 @@ parse-bbox-func-name=NvDsInferParseYoloE
 ...
 ```
 
+**NOTE**: The **YOLO-NAS** resizes the input with left/top padding. To get better accuracy, use
+
+```
+maintain-aspect-ratio=1
+symmetric-padding=0
+```
+
 ##
 
 ### Edit the deepstream_app_config file
@@ -168,4 +175,6 @@ config-file=config_infer_primary_yolonas.txt
 deepstream-app -c deepstream_app_config.txt
 ```
 
+**NOTE**: The TensorRT engine file may take a very long time to generate (sometimes more than 10 minutes).
+
 **NOTE**: For more information about custom models configuration (`batch-size`, `network-mode`, etc), please check the [`docs/customModels.md`](customModels.md) file.

docs/YOLOR.md

@@ -146,6 +146,13 @@ parse-bbox-func-name=NvDsInferParseYolo
 ...
 ```
 
+**NOTE**: The **YOLOR** resizes the input with center padding. To get better accuracy, use
+
+```
+maintain-aspect-ratio=1
+symmetric-padding=1
+```
+
 ##
 
 ### Edit the deepstream_app_config file
@@ -165,4 +172,6 @@ config-file=config_infer_primary_yolor.txt
 deepstream-app -c deepstream_app_config.txt
 ```
 
+**NOTE**: The TensorRT engine file may take a very long time to generate (sometimes more than 10 minutes).
+
 **NOTE**: For more information about custom models configuration (`batch-size`, `network-mode`, etc), please check the [`docs/customModels.md`](customModels.md) file.

docs/YOLOX.md

@@ -32,7 +32,7 @@ Copy the `export_yolox.py` file from `DeepStream-Yolo/utils` directory to the `Y
 
 #### 3. Download the model
 
-Download the `pth` file from [YOLOX](https://github.com/Megvii-BaseDetection/YOLOX/releases/) releases (example for YOLOX-s standard)
+Download the `pth` file from [YOLOX](https://github.com/Megvii-BaseDetection/YOLOX/releases/) releases (example for YOLOX-s)
 
 ```
 wget https://github.com/Megvii-BaseDetection/YOLOX/releases/download/0.1.1rc0/yolox_s.pth
@@ -42,7 +42,7 @@ wget https://github.com/Megvii-BaseDetection/YOLOX/releases/download/0.1.1rc0/yo
 
 #### 4. Convert model
 
-Generate the ONNX model file (example for YOLOX-s standard)
+Generate the ONNX model file (example for YOLOX-s)
 
 ```
 python3 export_yolox.py -w yolox_s.pth -c exps/default/yolox_s.py --simplify
@@ -98,7 +98,7 @@ Open the `DeepStream-Yolo` folder and compile the lib
 
 ### Edit the config_infer_primary_yolox file
 
-Edit the `config_infer_primary_yolox.txt` file according to your model (example for YOLOX-s standard with 80 classes)
+Edit the `config_infer_primary_yolox.txt` file according to your model (example for YOLOX-s with 80 classes)
 
 ```
 [property]
@@ -114,10 +114,17 @@ parse-bbox-func-name=NvDsInferParseYolo
 
 **NOTE**: If you use the **legacy** model, you should edit the `config_infer_primary_yolox_legacy.txt` file.
 
-**NOTE**: The **YOLOX standard** uses no normalization on the image preprocess. It is important to change the `net-scale-factor` according to the trained values.
+**NOTE**: The **YOLOX and YOLOX legacy** resize the input with left/top padding. To get better accuracy, use
 
 ```
-net-scale-factor=0
+maintain-aspect-ratio=1
+symmetric-padding=0
+```
+
+**NOTE**: The **YOLOX** uses no normalization on the image preprocess. It is important to change the `net-scale-factor` according to the trained values.
+
+```
+net-scale-factor=1
 ```
 
 **NOTE**: The **YOLOX legacy** uses normalization on the image preprocess. It is important to change the `net-scale-factor` and `offsets` according to the trained values.
@@ -150,4 +157,6 @@ config-file=config_infer_primary_yolox.txt
 deepstream-app -c deepstream_app_config.txt
 ```
 
+**NOTE**: The TensorRT engine file may take a very long time to generate (sometimes more than 10 minutes).
+
 **NOTE**: For more information about custom models configuration (`batch-size`, `network-mode`, etc), please check the [`docs/customModels.md`](customModels.md) file.

docs/YOLOv5.md

@@ -138,6 +138,13 @@ parse-bbox-func-name=NvDsInferParseYolo
 ...
 ```
 
+**NOTE**: The **YOLOv5** resizes the input with center padding. To get better accuracy, use
+
+```
+maintain-aspect-ratio=1
+symmetric-padding=1
+```
+
 ##
 
 ### Edit the deepstream_app_config file
@@ -157,4 +164,6 @@ config-file=config_infer_primary_yoloV5.txt
 deepstream-app -c deepstream_app_config.txt
 ```
 
+**NOTE**: The TensorRT engine file may take a very long time to generate (sometimes more than 10 minutes).
+
 **NOTE**: For more information about custom models configuration (`batch-size`, `network-mode`, etc), please check the [`docs/customModels.md`](customModels.md) file.

docs/YOLOv6.md

@@ -136,6 +136,13 @@ parse-bbox-func-name=NvDsInferParseYolo
 ...
 ```
 
+**NOTE**: The **YOLOv6** resizes the input with center padding. To get better accuracy, use
+
+```
+maintain-aspect-ratio=1
+symmetric-padding=1
+```
+
 ##
 
 ### Edit the deepstream_app_config file
@@ -155,4 +162,6 @@ config-file=config_infer_primary_yoloV6.txt
 deepstream-app -c deepstream_app_config.txt
 ```
 
+**NOTE**: The TensorRT engine file may take a very long time to generate (sometimes more than 10 minutes).
+
 **NOTE**: For more information about custom models configuration (`batch-size`, `network-mode`, etc), please check the [`docs/customModels.md`](customModels.md) file.

docs/YOLOv7.md

@@ -140,6 +140,13 @@ parse-bbox-func-name=NvDsInferParseYolo
 ...
 ```
 
+**NOTE**: The **YOLOv7** resizes the input with center padding. To get better accuracy, use
+
+```
+maintain-aspect-ratio=1
+symmetric-padding=1
+```
+
 ##
 
 ### Edit the deepstream_app_config file
@@ -159,4 +166,6 @@ config-file=config_infer_primary_yoloV7.txt
 deepstream-app -c deepstream_app_config.txt
 ```
 
+**NOTE**: The TensorRT engine file may take a very long time to generate (sometimes more than 10 minutes).
+
 **NOTE**: For more information about custom models configuration (`batch-size`, `network-mode`, etc), please check the [`docs/customModels.md`](customModels.md) file.

docs/YOLOv8.md

@@ -131,6 +131,13 @@ parse-bbox-func-name=NvDsInferParseYolo
 ...
 ```
 
+**NOTE**: The **YOLOv8** resizes the input with center padding. To get better accuracy, use
+
+```
+maintain-aspect-ratio=1
+symmetric-padding=1
+```
+
 ##
 
 ### Edit the deepstream_app_config file
@@ -150,4 +157,6 @@ config-file=config_infer_primary_yoloV8.txt
 deepstream-app -c deepstream_app_config.txt
 ```
 
+**NOTE**: The TensorRT engine file may take a very long time to generate (sometimes more than 10 minutes).
+
 **NOTE**: For more information about custom models configuration (`batch-size`, `network-mode`, etc), please check the [`docs/customModels.md`](customModels.md) file.