[INFORMATION] mmyolo - Model Surgery using edgeai-modeloptimization - to create lite models #7
Description
Introduction
mmyolo (https://github.com/open-mmlab/mmyolo) is a repository that has several interesting Object Detection models. For example, it includes models such as YOLOv5, YOLOv7, YOLOX, YOLOv8 etc.
Here we describe how to apply Model Surgery on mmyolo to create lite models that run faster on Embedded Systems.
Background - What actually happens in Model Surgery
The types of Operators/Layers that are being used in popular models are increasing rapidly. All of them may not work efficiently in embedded devices. For example, a ReLU activation layer is much faster compared to a SWish activation layer - because ReLU operator is implemented in Hardware at fullest speed (because of the simplicity of ReLU operation). This is just an example. There are several such examples.
In many cases it is possible to replace in-efficient layers with their efficient alternatives without actually modifying the code. It is done by modifying the Python model after the model has been instantiated.
How to use edgeai-modeloptimization
edgeai-modeloptimization (https://github.com/TexasInstruments/edgeai-tensorlab/tree/main/edgeai-modeloptimization) is a package that can automate some of the Model Surgery aspects.
It provides edgeai_torchmodelopt, a python pakage that helps to modify PyTorch models without manually editing the model code.
The exact location is here: https://github.com/TexasInstruments/edgeai-tensorlab/tree/main/edgeai-modeloptimization/torchmodelopt
It provides various types of model surgery options as described here:
https://github.com/TexasInstruments/edgeai-tensorlab/blob/main/edgeai-modeloptimization/torchmodelopt/docs/surgery.md
Patch file
The commit id of mmyolo (https://github.com/open-mmlab/mmyolo) for this explanation is: 8c4d9dc503dc8e327bec8147e8dc97124052f693
This patch file includes above modification in train.py and other modifications in val.py, prototxt export etc.
0001-2024-Aug-2-mmyolo.commit-8c4d9dc5.-model-surgery-with-edgeai-modeloptimization.txt
Patching mmyolo:
git clone https://github.com/open-mmlab/mmyolo.git
git checkout 8c4d9dc5
git am 0001-mmyolo.commit-8c4d9dc5.-model-surgery-with-edgeai-modeloptimization.txt
Run training:
python3 tools/train.py <configfile> --model-surgery 1
You can also use tools/dist_train.sh
(just make sure that --model-surgery 1 argument is passed inside it)
Expected Accuracy
This table shows expected model accuracy of Lite models after training.
| Dataset | Original Model | Lite Model | Input Size | Original AP[0.5:0.95]%, AP50% | Lite AP[0.5:0.95]%, AP50% | GigaMACS | config file | Notes |
|---|---|---|---|---|---|---|---|---|
| YOLOv5 models | ||||||||
| COCO | YOLOv5-nano | YOLOv5-nano-lite | 640x640 | 28.0, 45.9 | 25.2, 42.1 | 2.07 | configs/yolov5/yolov5_n-v61_syncbn_fast_8xb16-300e_coco.py | |
| COCO | YOLOv5-small | YOLOv5-small-lite | 640x640 | 37.7, 57.1 | 35.5, 54.7 | 7.89 | configs/yolov5/yolov5_s-v61_syncbn_fast_8xb16-300e_coco.py | |
| YOLOv7 models | ||||||||
| COCO | YOLOv7-tiny | YOLOv7-tiny-lite | 640x640 | 37.5, 55.8 | 36.7, 55.0 | 6.87 | configs/yolov7/yolov7_tiny_syncbn_fast_8x16b-300e_coco.py | |
| COCO | YOLOv7-large | YOLOv7-large-lite | 640x640 | 51.0, 69.0 | 48.1, 66.4 | 52.95 | configs/yolov7/yolov7_l_syncbn_fast_8x16b-300e_coco.py | |
| YOLOv8 models | ||||||||
| COCO | YOLOv8-nano | YOLOv8-nano-lite | 640x640 | 37.2, 52.7 | 34.5, 49.7 | - | configs/yolov8/yolov8_n_syncbn_fast_8xb16-500e_coco.py | |
| COCO | YOLOv8-small | YOLOv8-small-lite | 640x640 | 44.2, 61.0 | 42.4, 58.8 | 14.33 | configs/yolov8/yolov8_s_syncbn_fast_8xb16-500e_coco.py | |
| YOLOX models | ||||||||
| COCO | YOLOX-tiny | YOLOX-tiny-lite | 416x416 | 32.7, 50.3 | 31.1, 48.4 | 3.25 | configs/yolox/yolox_tiny_fast_8xb8-300e_coco.py | |
| COCO | YOLOX-small | YOLOX-small-lite | 640x640 | 40.7, 59.6 | 38.7, 57.4 | 7.85 | configs/yolox/yolox_s_fast_8xb8-300e_coco.py |
Notes
- GigaMACS: Complexity in Giga Multiply-Accumulations required for inference (lower is better). This is an important metric to watch out for when selecting models for embedded inference.
- Accuracy for Object Detection on COCO dataset primarily uses two accuracy metrics AP[0.5:0.95] and AP50 (in percentages). AP[0.5:0.95] is the Mean of Average Precision values computed at IOUs ranging from 0.5 to 0.95 and averaged. AP50 is the Average Precision computed at 0.5 IoU. If only one accuracy metric is mentioned in a table cell, then it is AP[0.5:0.95]. Be sure to compare using the same metric when comparing across various detectors or configurations.
- Input size in the tables (width x height) indicates the resolution for the model input. Original input images can be resized to that resolution with preserving the aspect ratio (may need padding) or without preserving the aspect ratio (depending on the flag keep_ratio in config files).
Additional information
Additional information about the details of the modifications done using Model Surgery is here: https://github.com/TexasInstruments/edgeai-yolov5