Training ends at a particular step

[sivakishoresk]

Hello,
I'm trying to replicate the entire training process, it seems it's getting crashed everytime in the 5th iteration. Could you please help me fix this.

Here is the Error trace including the logs of step 3 :

step:3 - timing/gen:305.090 - timing/verify:8.531 - train_verify_score/math:0.381 - train_verify_score/all:0.381 - reward_model/prm_loss:0.720 - reward_model/grad_norm:5.092 - reward_model/dpo_acc_before:0.581 - timing/reward_model:107.155 - train_reward/reward_model:0.023 - train_reward/verifier:0.492 - train_reward/reward_all:2.576 - critic/kl:0.000 - critic/kl_coeff:0.000 - timing/adv:1.017 - timing/update_actor:84.257 - actor/pg_loss:-0.650 - actor/pg_clipfrac:0.002 - actor/ppo_kl:0.000 - actor/grad_norm:0.158 - actor/lr(1e-4):0.005 - critic/score/mean:2.576 - critic/score/max:9.550 - critic/score/min:-4.858 - critic/rewards/mean:2.576 - critic/rewards/max:9.550 - critic/rewards/min:-4.858 - critic/advantages/mean:-0.000 - critic/advantages/max:3.169 - critic/advantages/min:-2.927 - critic/returns/mean:-0.048 - critic/returns/max:2.113 - critic/returns/min:-2.044 - response_length/mean:1435.491 - response_length/max:3072.000 - response_length/min:73.000 - prompt_length/mean:193.648 - prompt_length/max:445.000 - prompt_length/min:141.000
(main_task pid=1367807) logging3: 0.0 seconds
(main_task pid=1367807) 264
(main_task pid=1367807) gen: 97.8 seconds
(main_task pid=1367807) WARNING:2025-02-20 07:19:44,376:WARNING: Error in configuration: macro '\frac' failed its substitution!
(main_task pid=1367807) WARNING:2025-02-20 07:19:44,567:WARNING: Error in configuration: macro '\frac' failed its substitution!
(main_task pid=1367807) Accuracy distribution: 0.00:136 0.25:31 0.50:24 0.75:18 1.00:55
(main_task pid=1367807) Filtered batch size: 292 (from original size: 1056)
(main_task pid=1367807) verify: 2.4 seconds
(main_task pid=1367807) collected 292 / 1024 rollouts and each prompt has 4 responses
(main_task pid=1367807) gen: 97.7 seconds
(main_task pid=1367807) WARNING:2025-02-20 07:21:24,810:WARNING: Error in configuration: macro '\frac' failed its substitution!
(main_task pid=1367807) Accuracy distribution: 0.00:110 0.25:29 0.50:22 0.75:22 1.00:73
(main_task pid=1367807) Filtered batch size: 292 (from original size: 1024)
(main_task pid=1367807) verify: 3.7 seconds
(main_task pid=1367807) collected 584 / 1024 rollouts and each prompt has 4 responses
Error executing job with overrides: ['data.train_files=[/data/siva.gollapalli/AIMO_experiments/PRIME/datasets/RL_data/subset_data/train.parquet]', 'data.val_files=[/data/siva.gollapalli/AIMO_experiments/PRIME/datasets/RL_data/subset_data/validation.parquet]', 'data.train_batch_size=256', 'data.val_batch_size=2', 'data.max_prompt_length=1024', 'data.max_response_length=3072', 'actor_rollout_ref.model.path=/data/siva.gollapalli/AIMO_experiments/PRIME/models/models--PRIME-RL--Eurus-2-7B-SFT/snapshots/3c18b631d89f80876bffb4db2ef0e8a989717a7c', 'actor_rollout_ref.actor.optim.lr=5e-7', 'actor_rollout_ref.actor.ppo_mini_batch_size=256', 'actor_rollout_ref.actor.ppo_micro_batch_size=8', 'actor_rollout_ref.actor.fsdp_config.param_offload=True', 'actor_rollout_ref.actor.fsdp_config.grad_offload=True', 'actor_rollout_ref.actor.fsdp_config.optimizer_offload=True', 'actor_rollout_ref.model.use_remove_padding=False', 'actor_rollout_ref.actor.entropy_coeff=0.', 'actor_rollout_ref.rollout.log_prob_micro_batch_size=64', 'actor_rollout_ref.rollout.tensor_model_parallel_size=1', 'actor_rollout_ref.rollout.name=vllm', 'actor_rollout_ref.rollout.gpu_memory_utilization=0.7', 'actor_rollout_ref.ref.log_prob_micro_batch_size=64', 'actor_rollout_ref.ref.fsdp_config.param_offload=True', 'algorithm.kl_ctrl.kl_coef=0.00', 'trainer.logger=[console,wandb]', 'trainer.project_name=testing', 'trainer.experiment_name=online-after-solvable-0.2-0.8-policy-self-ref', 'trainer.default_local_dir=/data/siva.gollapalli/AIMO_experiments/PRIME/RL_training/testing/online-after-solvable-0.2-0.8-policy-self-ref', 'trainer.n_gpus_per_node=8', 'trainer.nnodes=1', 'trainer.save_freq=16', 'trainer.test_freq=16', 'trainer.total_epochs=1', 'data.n_samples=4', 'data.filter_accuracy=True', 'data.accuracy_lower_bound=0.2', 'data.accuracy_upper_bound=0.8', 'algorithm.adv_estimator=rloo', 'algorithm.adv_params.verifier_gamma=1.0', 'algorithm.adv_params.reward_model_gamma=1.0', 'reward_model.rm_type=prime', 'reward_model.rm_coef=5', 'reward_model.prime_model.path=/data/siva.gollapalli/AIMO_experiments/PRIME/models/models--PRIME-RL--Eurus-2-7B-SFT/snapshots/3c18b631d89f80876bffb4db2ef0e8a989717a7c', 'reward_model.prime_model.ref_path=/data/siva.gollapalli/AIMO_experiments/PRIME/models/models--PRIME-RL--Eurus-2-7B-SFT/snapshots/3c18b631d89f80876bffb4db2ef0e8a989717a7c', 'reward_model.model.input_tokenizer=null', 'reward_model.prime_model.use_remove_padding=False', 'reward_model.prime_granularity=token', 'reward_model.micro_batch_size=8', 'reward_model.prime_model.update=before', 'reward_model.prime_model.beta_train=0.05', 'reward_model.prime_model.optim.lr=1e-6', 'reward_model.prime_model.optim.grad_clip=10.0', 'reward_model.prime_model.input_tokenizer=null', 'trainer.default_local_dir=/data/siva.gollapalli/AIMO_experiments/PRIME/RL_training/testing/online-after-solvable-0.2-0.8-policy-self-ref']
Traceback (most recent call last):
File "/data/siva.gollapalli/AIMO_experiments/PRIME/PRIME/training/verl/trainer/main_ppo.py", line 131, in main
ray.get(main_task.remote(config))
File "/data/siva.gollapalli/AIMO_experiments/PRIME/prime_train_env/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
return fn(*args, **kwargs)
File "/data/siva.gollapalli/AIMO_experiments/PRIME/prime_train_env/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
File "/data/siva.gollapalli/AIMO_experiments/PRIME/prime_train_env/lib/python3.10/site-packages/ray/_private/worker.py", line 2772, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
File "/data/siva.gollapalli/AIMO_experiments/PRIME/prime_train_env/lib/python3.10/site-packages/ray/_private/worker.py", line 919, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(IndexError): ray::main_task() (pid=1367807, ip=10.67.28.2)
File "/data/siva.gollapalli/AIMO_experiments/PRIME/PRIME/training/verl/trainer/main_ppo.py", line 223, in main_task
trainer.fit()
File "/data/siva.gollapalli/AIMO_experiments/PRIME/PRIME/training/verl/trainer/ppo/ray_trainer.py", line 626, in fit
val_metrics = self._validate()
File "/data/siva.gollapalli/AIMO_experiments/PRIME/PRIME/training/verl/trainer/ppo/ray_trainer.py", line 321, in _validate
test_output_gen_batch = self.actor_rollout_wg.generate_sequences(test_gen_batch)
File "/data/siva.gollapalli/AIMO_experiments/PRIME/PRIME/training/verl/single_controller/ray/base.py", line 39, in func
args, kwargs = dispatch_fn(self, *args, **kwargs)
File "/data/siva.gollapalli/AIMO_experiments/PRIME/PRIME/training/verl/single_controller/base/decorator.py", line 275, in dispatch_dp_compute_data_proto
splitted_args, splitted_kwargs = _split_args_kwargs_data_proto(worker_group.world_size, *args, **kwargs)
File "/data/siva.gollapalli/AIMO_experiments/PRIME/PRIME/training/verl/single_controller/base/decorator.py", line 50, in _split_args_kwargs_data_proto
splitted_args.append(arg.chunk(chunks=chunks))
File "/data/siva.gollapalli/AIMO_experiments/PRIME/PRIME/training/verl/protocol.py", line 415, in chunk
DataProto(batch=batch_lst[i], non_tensor_batch=non_tensor_batch_lst[i], meta_info=self.meta_info))
IndexError: tuple index out of range

[sivakishoresk]

This is the config file with which the error occured

set -x

export NCCL_DEBUG=WARN
export WANDB_API_KEY=''
export VLLM_ATTENTION_BACKEND=XFORMERS
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export TOKENIZERS_PARALLELISM=true

PROJECT_NAME='testing'
EXPERIMENT_NAME='online-after-solvable-0.2-0.8-policy-self-ref'
DATA_PATH="/data/siva.gollapalli/AIMO_experiments/PRIME/datasets/RL_data/subset_data"
SFT_MODEL_PATH=/data/siva.gollapalli/AIMO_experiments/PRIME/models/models--PRIME-RL--Eurus-2-7B-SFT/snapshots/3c18b631d89f80876bffb4db2ef0e8a989717a7c
CKPT_PATH=/data/siva.gollapalli/AIMO_experiments/PRIME/RL_training

python3 -m verl.trainer.main_ppo
data.train_files=["$DATA_PATH/train.parquet"]
data.val_files=["$DATA_PATH/validation.parquet"]
data.train_batch_size=256
data.val_batch_size=2
data.max_prompt_length=1024
data.max_response_length=3072
actor_rollout_ref.model.path=$SFT_MODEL_PATH
actor_rollout_ref.actor.optim.lr=5e-7
actor_rollout_ref.actor.ppo_mini_batch_size=256
actor_rollout_ref.actor.ppo_micro_batch_size=8
actor_rollout_ref.actor.fsdp_config.param_offload=True
actor_rollout_ref.actor.fsdp_config.grad_offload=True
actor_rollout_ref.actor.fsdp_config.optimizer_offload=True
actor_rollout_ref.model.use_remove_padding=True
actor_rollout_ref.actor.entropy_coeff=0.
actor_rollout_ref.rollout.log_prob_micro_batch_size=64
actor_rollout_ref.rollout.tensor_model_parallel_size=1
actor_rollout_ref.rollout.name=vllm
actor_rollout_ref.rollout.gpu_memory_utilization=0.7
actor_rollout_ref.ref.log_prob_micro_batch_size=64
actor_rollout_ref.ref.fsdp_config.param_offload=True
algorithm.kl_ctrl.kl_coef=0.00
trainer.logger=['console','wandb']
trainer.project_name=$PROJECT_NAME
trainer.experiment_name=$EXPERIMENT_NAME
trainer.default_local_dir=$CKPT_PATH/$PROJECT_NAME/$EXPERIMENT_NAME
trainer.n_gpus_per_node=8
trainer.nnodes=1
trainer.save_freq=16
trainer.test_freq=16
trainer.total_epochs=1
data.n_samples=4
data.filter_accuracy=True
data.accuracy_lower_bound=0.2
data.accuracy_upper_bound=0.8
algorithm.adv_estimator=rloo
algorithm.adv_params.verifier_gamma=1.0
algorithm.adv_params.reward_model_gamma=1.0
reward_model.rm_type=prime
reward_model.rm_coef=5
reward_model.prime_model.path=$SFT_MODEL_PATH
reward_model.prime_model.ref_path=$SFT_MODEL_PATH
reward_model.model.input_tokenizer=null
reward_model.prime_model.use_remove_padding=True
reward_model.prime_granularity=token
reward_model.micro_batch_size=8
reward_model.prime_model.update=before
reward_model.prime_model.beta_train=0.05
reward_model.prime_model.optim.lr=1e-6
reward_model.prime_model.optim.grad_clip=10.0
reward_model.prime_model.input_tokenizer=null
trainer.default_local_dir=$CKPT_PATH/$PROJECT_NAME/$EXPERIMENT_NAME \

[sivakishoresk]

The following is the environment I'm using :

accelerate==1.3.0
aiohappyeyeballs==2.4.6
aiohttp==3.11.12
aiosignal==1.3.2
annotated-types==0.7.0
antlr4-python3-runtime==4.9.3
anyio==4.8.0
async-timeout==5.0.1
attrs==25.1.0
certifi==2025.1.31
charset-normalizer==3.4.1
click==8.1.8
cloudpickle==3.1.1
codetiming==1.4.0
datasets==3.2.0
dill==0.3.8
diskcache==5.6.3
distro==1.9.0
docker-pycreds==0.4.0
einops==0.8.1
exceptiongroup==1.2.2
fastapi==0.115.8
filelock==3.17.0
flash_attn==2.7.4.post1
frozenlist==1.5.0
fsspec==2024.9.0
gguf==0.10.0
gitdb==4.0.12
GitPython==3.1.44
h11==0.14.0
httpcore==1.0.7
httptools==0.6.4
httpx==0.28.1
huggingface-hub==0.28.1
hydra-core==1.3.2
idna==3.10
importlib_metadata==8.6.1
interegular==0.3.3
Jinja2==3.1.5
jiter==0.8.2
jsonschema==4.23.0
jsonschema-specifications==2024.10.1
lark==1.2.2
liger_kernel==0.5.2
llvmlite==0.44.0
lm-format-enforcer==0.10.6
MarkupSafe==3.0.2
mistral_common==1.5.3
mpmath==1.3.0
msgpack==1.1.0
msgspec==0.19.0
multidict==6.1.0
multiprocess==0.70.16
nest-asyncio==1.6.0
networkx==3.4.2
numba==0.61.0
numpy==1.26.4
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==9.1.0.70
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-ml-py==12.570.86
nvidia-nccl-cu12==2.20.5
nvidia-nvjitlink-cu12==12.8.61
nvidia-nvtx-cu12==12.1.105
omegaconf==2.3.0
openai==1.61.1
opencv-python-headless==4.11.0.86
orjson==3.10.15
outlines==0.0.46
packaging==24.2
pandas==2.2.3
partial-json-parser==0.2.1.1.post5
peft==0.14.0
pillow==11.1.0
platformdirs==4.3.6
prometheus-fastapi-instrumentator==7.0.2
prometheus_client==0.21.1
propcache==0.2.1
protobuf==5.29.3
psutil==6.1.1
py-cpuinfo==9.0.0
pyairports==2.1.1
pyarrow==19.0.0
pybind11==2.13.6
pycountry==24.6.1
pydantic==2.10.6
pydantic_core==2.27.2
pyext==0.7
pylatexenc==2.10
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
pytz==2025.1
PyYAML==6.0.2
pyzmq==26.2.1
ray==2.42.0
referencing==0.36.2
regex==2024.11.6
requests==2.32.3
rpds-py==0.22.3
safetensors==0.5.2
sentencepiece==0.2.0
sentry-sdk==2.20.0
setproctitle==1.3.4
six==1.17.0
smmap==5.0.2
sniffio==1.3.1
starlette==0.45.3
sympy==1.13.3
tensordict==0.5.0
tiktoken==0.8.0
tokenizers==0.21.0
torch==2.4.0
torchvision==0.19.0
tqdm==4.67.1
transformers==4.47.1
triton==3.0.0
typing_extensions==4.12.2
tzdata==2025.1
urllib3==2.3.0
uvicorn==0.34.0
uvloop==0.21.0
-e git+https://github.com/volcengine/verl.git@3d566adc1604ccae1ad3523a878d0733d5be3df5#egg=verl
vllm==0.6.3
wandb==0.19.6
watchfiles==1.0.4
websockets==14.2
xformers==0.0.27.post2
xxhash==3.5.0
yarl==1.18.3
zipp==3.21.0

[cgq15]

The data.val_batch_size can't be smaller than the world size (2 and 8 here)
Maybe you can try a larger val_batch_size, at least 8 :)