[edgeai-modelmaker] Basic command ./run_modelmaker.sh AM69A config_detection.yaml does not work on v 11.0, 10.1 #20
Description
I cannot run even the most basic command in this repo - train and compile model for pre-configured detection. I target SDK Version 11, this is what's installed on my dev AM69A
Please advise what to do.
Command
./run_modelmaker.sh AM69A config_detection.yaml
Error in the log
...
TASKS TOTAL=1, NUM_RUNNING=0: 100%|████████████████████████████| 1/1 [00:03<00:00, 4.24it/s, postfix={'RUNNING': [], 'COMPLETED': ['od-8200']}]
WARNING: Benchmark - completed: 0/1
TASKS TOTAL=1, NUM_RUNNING=0: 100%|████████████████████████████| 1/1 [00:04<00:00, 4.06s/it, postfix={'RUNNING': [], 'COMPLETED': ['od-8200']}]
INFO: packaging artifacts to /workspace/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20250725-233321/yolox_nano_lite/compilation/pkg please wait...
WARNING:20250725-233839: could not package - /workspace/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20250725-233321/yolox_nano_lite/compilation/work/od-8200
Traceback (most recent call last):
File "/workspace/edgeai-tensorlab/edgeai-modelmaker/./scripts/run_modelmaker.py", line 153, in <module>
main(config)
File "/workspace/edgeai-tensorlab/edgeai-modelmaker/./scripts/run_modelmaker.py", line 88, in main
model_runner.run()
File "/workspace/edgeai-tensorlab/edgeai-modelmaker/edgeai_modelmaker/ai_modules/vision/runner.py", line 222, in run
self.model_compilation.run()
File "/workspace/edgeai-tensorlab/edgeai-modelmaker/edgeai_modelmaker/ai_modules/vision/compilation/edgeai_benchmark.py", line 164, in run
edgeai_benchmark.interfaces.package_artifacts(self.settings, self.work_dir, out_dir=self.package_dir, custom_model=True)
File "/workspace/edgeai-tensorlab/edgeai-benchmark/edgeai_benchmark/interfaces/run_package.py", line 271, in package_artifacts
with open(os.path.join(out_dir,'artifacts.yaml'), 'w') as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/workspace/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20250725-233321/yolox_nano_lite/compilation/pkg/artifacts.yaml'
Environment
Host - Ubuntu 22.4 with NVIDIA 4090 and 4070, NVIDIA-SMI 535.129.03 Driver Version: 535.129.03 CUDA Version: 12.2
Dockerfile:
FROM nvidia/cuda:12.1.0-cudnn8-devel-ubuntu22.04
ENV DEBIAN_FRONTEND=noninteractive
# Install system dependencies
RUN apt-get update && apt-get install -y \
python3.10 \
python3.10-dev \
python3-pip \
git \
wget \
unzip \
libgl1 \
libglib2.0-0 \
libsm6 \
libxext6 \
libxrender-dev \
libgomp1 \
libglib2.0-0 \
gcc \
g++ \
ninja-build \
&& rm -rf /var/lib/apt/lists/*
RUN python3 -m pip install --upgrade pip
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.10 1
WORKDIR /workspace
RUN git clone https://github.com/TexasInstruments/edgeai-tensorlab.git && \
cd edgeai-tensorlab && \
git checkout r11.0
RUN pip3 install torch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 --index-url https://download.pytorch.org/whl/cu121
RUN mkdir -p /workspace/datasets \
/workspace/outputs \
/workspace/models \
/workspace/data/downloads \
/workspace/data/projects
ENV PYTHONPATH="/workspace/edgeai-tensorlab/edgeai-modelmaker:/workspace/edgeai-tensorlab/edgeai-benchmark:/workspace/edgeai-tensorlab/edgeai-mmdetection:/workspace/edgeai-tensorlab:${PYTHONPATH}"
WORKDIR /workspace/edgeai-tensorlab/edgeai-modelmaker
RUN ./setup_gpu.sh
CMD ["/bin/bash"]
Full log
./run_modelmaker.sh AM69A config_detection.yaml
Number of AVX cores detected in PC: 32
AVX compilation speedup in PC : 1
Target device : AM69A
PYTHONPATH : .:/workspace/edgeai-tensorlab/edgeai-modelmaker:/workspace/edgeai-tensorlab/edgeai-benchmark:/workspace/edgeai-tensorlab/edgeai-mmdetection:/workspace/edgeai-tensorlab:
TIDL_TOOLS_PATH : ../edgeai-benchmark/tools/tidl_tools_package/AM69A/tidl_tools
LD_LIBRARY_PATH : ../edgeai-benchmark/tools/tidl_tools_package/AM69A/tidl_tools:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
argv: ['./scripts/run_modelmaker.py', 'config_detection.yaml', '--target_device', 'AM69A']
---------------------------------------------------------------------
INFO: ModelMaker - task_type:detection model_name:yolox_nano_lite dataset_name:tiscapes2017_driving run_name:20250725-233321/yolox_nano_lite
- Model: yolox_nano_lite
- TargetDevices & Estimated Inference Times (ms): {'TDA4VM': 3.74, 'AM62A': 8.87, 'AM67A': '8.87 (with 1/2 device capability)', 'AM68A': 3.73, 'AM69A': '3.64 (with 1/4th device capability)', 'AM62': 516.15}
- This model can be compiled for the above device(s).
---------------------------------------------------------------------
downloading from https://software-dl.ti.com/jacinto7/esd/modelzoo/10_01_00/models/vision/detection/coco/edgeai-mmdet/yolox_nano_lite_416x416_20220214_checkpoint.pth to ./data/downloads/pretrained/yolox_nano_lite/yolox_nano_lite_416x416_20220214_checkpoint.pth
100%|███████████████████████████████████████████████████████████████████████████████████████████| 27605950/27605950 [00:06<00:00, 4015555.32B/s]
downloading from http://software-dl.ti.com/jacinto7/esd/modelzoo/08_06_00_01/datasets/tiscapes2017_driving.zip to ./data/downloads/datasets/tiscapes2017_driving.zip
100%|█████████████████████████████████████████████████████████████████████████████████████████| 113880250/113880250 [00:19<00:00, 5848067.30B/s]
INFO: ModelMaker - dataset split sizes {'train': 393, 'val': 107}
INFO: ModelMaker - max_num_files is set to: 10000
INFO: ModelMaker - dataset split sizes are limited to: {'train': 393, 'val': 107}
INFO: ModelMaker - dataset loading OK
loading annotations into memory...
Done (t=0.09s)
creating index...
index created!
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
WARNING:20250725-233350: model_shortlist=100 - this will cause only a subset of models to be selected for run
WARNING:20250725-233350: if the model that you wish is not being selected for run, then remove this model_shortlist -
WARNING:20250725-233350: this model_shortlist could be being set in settings_base.yaml or passed inside run_benchmarks_pc.sh -
INFO: ModelMaker - run params is at: /workspace/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20250725-233321/yolox_nano_lite/run.yaml
INFO: ModelMaker - running training - for detailed info see the log file: /workspace/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20250725-233321/yolox_nano_lite/training/run.log
TASKS TOTAL=1, NUM_RUNNING=0: 100%|███████████████████| 1/1 [04:45<00:00, 285.18s/it, postfix={'RUNNING': [], 'COMPLETED': ['yolox_nano_lite']}]
Trained model is at: /workspace/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20250725-233321/yolox_nano_lite/training
SUCCESS: ModelMaker - Training completed.
INFO: ModelMaker - running compilation - for detailed info see the log file: /workspace/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20250725-233321/yolox_nano_lite/compilation/work/od-8200/run.log
WARNING:20250725-233835: model_shortlist=100 - this will cause only a subset of models to be selected for run
WARNING:20250725-233835: if the model that you wish is not being selected for run, then remove this model_shortlist -
WARNING:20250725-233835: this model_shortlist could be being set in settings_base.yaml or passed inside run_benchmarks_pc.sh -
INFO:20250725-233835: number of configs - 1
TASKS TOTAL=1, NUM_RUNNING=1: 0%| | 0/1 [00:01<?, ?it/s, postfix={'RUNNING': ['od-8200:import'], 'COMPLETED': []}]
ERROR:20250725-233837: Error occurred: od-8200:import - Error Code: -11 at /workspace/edgeai-tensorlab/edgeai-benchmark/edgeai_benchmark/utils/parallel_runner.py
TASKS TOTAL=1, NUM_RUNNING=1: 0%| | 0/1 [00:03<?, ?it/s, postfix={'RUNNING': ['od-8200:infer'], 'COMPLETED': []}]
ERROR:20250725-233839: Error occurred: od-8200:infer - Error Code: -11 at /workspace/edgeai-tensorlab/edgeai-benchmark/edgeai_benchmark/utils/parallel_runner.py
TASKS TOTAL=1, NUM_RUNNING=0: 100%|████████████████████████████| 1/1 [00:03<00:00, 4.24it/s, postfix={'RUNNING': [], 'COMPLETED': ['od-8200']}]
WARNING: Benchmark - completed: 0/1
TASKS TOTAL=1, NUM_RUNNING=0: 100%|████████████████████████████| 1/1 [00:04<00:00, 4.06s/it, postfix={'RUNNING': [], 'COMPLETED': ['od-8200']}]
INFO: packaging artifacts to /workspace/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20250725-233321/yolox_nano_lite/compilation/pkg please wait...
WARNING:20250725-233839: could not package - /workspace/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20250725-233321/yolox_nano_lite/compilation/work/od-8200
Traceback (most recent call last):
File "/workspace/edgeai-tensorlab/edgeai-modelmaker/./scripts/run_modelmaker.py", line 153, in <module>
main(config)
File "/workspace/edgeai-tensorlab/edgeai-modelmaker/./scripts/run_modelmaker.py", line 88, in main
model_runner.run()
File "/workspace/edgeai-tensorlab/edgeai-modelmaker/edgeai_modelmaker/ai_modules/vision/runner.py", line 222, in run
self.model_compilation.run()
File "/workspace/edgeai-tensorlab/edgeai-modelmaker/edgeai_modelmaker/ai_modules/vision/compilation/edgeai_benchmark.py", line 164, in run
edgeai_benchmark.interfaces.package_artifacts(self.settings, self.work_dir, out_dir=self.package_dir, custom_model=True)
File "/workspace/edgeai-tensorlab/edgeai-benchmark/edgeai_benchmark/interfaces/run_package.py", line 271, in package_artifacts
with open(os.path.join(out_dir,'artifacts.yaml'), 'w') as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/workspace/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20250725-233321/yolox_nano_lite/compilation/pkg/artifacts.yaml'
File that was mentioned in the output
$ cat /workspace/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20250725-233321/yolox_nano_lite/compilation/work/od-8200/run.log
INFO:20250725-233836: starting - od-8200
INFO:20250725-233836: running - od-8200
INFO:20250725-233836: pipeline_config - {'task_type': 'detection', 'dataset_category': 'coco', 'calibration_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x7fb2576449a0>, 'input_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x7fb36b6bf130>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0x7fb255507a90>, 'session': <edgeai_benchmark.sessions.onnxrt_session.ONNXRTSession object at 0x7fb255507a30>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0x7fb2555042b0>, 'metric': {'label_offset_pred': 1}, 'model_info': {'metric_reference': {'accuracy_ap[.5:.95]%': None}, 'model_shortlist': 10, 'compact_name': 'yolox-nano-lite-mmdet-coco-416x416', 'shortlisted': True, 'recommended': True}}
INFO:20250725-233836: import - od-8200 - this may take some time...
INFO:20250725-233836: model_path - /workspace/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20250725-233321/yolox_nano_lite/training/model.onnx
INFO:20250725-233836: model_file - /workspace/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20250725-233321/yolox_nano_lite/compilation/work/od-8200/model/model.onnx
INFO:20250725-233836: quant_file - /workspace/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20250725-233321/yolox_nano_lite/compilation/work/od-8200/model/model_qparams.prototxt
Downloading 1/1: /workspace/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20250725-233321/yolox_nano_lite/training/model.onnx
Download done for /workspace/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20250725-233321/yolox_nano_lite/training/model.onnx
Downloading 1/1: /workspace/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20250725-233321/yolox_nano_lite/training/model.onnx
Download done for /workspace/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20250725-233321/yolox_nano_lite/training/model.onnx
INFO:20250725-233837: starting - od-8200
INFO:20250725-233837: running - od-8200
INFO:20250725-233837: pipeline_config - {'task_type': 'detection', 'dataset_category': 'coco', 'calibration_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x7fb2576449a0>, 'input_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x7fb36b6bf130>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0x7fb255507a90>, 'session': <edgeai_benchmark.sessions.onnxrt_session.ONNXRTSession object at 0x7fb255507a30>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0x7fb2555042b0>, 'metric': {'label_offset_pred': 1}, 'model_info': {'metric_reference': {'accuracy_ap[.5:.95]%': None}, 'model_shortlist': 10, 'compact_name': 'yolox-nano-lite-mmdet-coco-416x416', 'shortlisted': True, 'recommended': True}}
INFO:20250725-233837: infer - od-8200 - this may take some time...
INFO:20250725-233837: model_path - /workspace/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20250725-233321/yolox_nano_lite/training/model.onnx
INFO:20250725-233837: model_file - /workspace/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20250725-233321/yolox_nano_lite/compilation/work/od-8200/model/model.onnx
INFO:20250725-233837: quant_file - /workspace/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20250725-233321/yolox_nano_lite/compilation/work/od-8200/model/model_qparams.prototxt