Skip to content

Pipeline overview

Part 2: The Training Pipeline

Everything from here on describes how the models in Part 1 were made. If you can use those models as they are, you do not need this part. Read on if you want a translator for a different language pair, a translator tuned to a domain where generic quality not enough, or a model preference-optimized for some entirely different task for which you can write down a scoring function.

The code lives in the GaMS-Team/GaMS-Translator repository and is written for the LEONARDO EuroHPC supercomputer (SLURM + Singularity). It runs on other SLURM clusters (given the strong enough hardware) with minor changes, mainly to partition names and time limits. Where LEONARDO's specifics matter, this book says so.

Why three stages

The pipeline is three consecutive training stages, and each one exists because the previous one cannot do its job:

GaMS3-12B ──SFT──> merge ──> GaMS3-12B-SFT-Translator
                                      │
                                      ├──> generate translations ──> DPO pairs
                                      │
                                      └──DPO──> merge ──> GaMS3-12B-DPO-Translator
                                                                    │
                                                                    └──GRPO──> merge ──> GaMS3-12B-GRPO-Translator

SFT teaches the task and the format. A pretrained GaMS model does not know the chat template and has no notion that a user turn containing a Slovene instruction and an English document should be answered with a Slovene translation and nothing else. A thousand high-quality examples fix that.

DPO fixes the specific things the SFT model gets wrong. After SFT the model translates competently and formats badly: it collapses paragraph separators, it keeps generating past the end of the document, it leaves dates in English form. These are systematic, identifiable errors, which makes them the ideal target for preference learning. For each of them we can take the model's own output and repair it programmatically, producing a pair where the difference between chosen and rejected is exactly the behaviour we want to change. This is what on-policy means in practice, and it is the reason the DPO stage needs only about 1,500 pairs to be effective. If you observe an error in published models that can be systematically corrected, you can quickly fix it with DPO stage.

GRPO improves translation quality itself. DPO learns from a fixed set of pairs, which clearly show the difference between the desired and bad response. Translation quality is not like that. It is difficult to turn a mediocre translation into a good one systematically. Instead, we can estimate the quality of each translation through reference-free metrics such as CometKiwi. Using GRPO, we can ask the model to translate the same example multiple times, score each translation and turn scores into a gradient through a reward function. GRPO pushes the model toward the candidates (translations) with higher scores than their group's average. Unlike DPO, the samples are drawn fresh from the current policy at every step, which is why this stage is the expensive one.

Stage map

# Stage Directory Nodes × GPUs Wall time
1 Build the Singularity container singularity/ login node minutes
2 Download models, reward models and GRPO data - 1 (serial, no GPU) up to 2 h
3 SFT training sft/ 1 × 4 8 h limit
4 Merge LoRA adapter merge_lora/ 1 × 1 30 min limit
5 Generate on-policy translations for DPO dpo/get_translations/ 1 × 4 2 h limit
6 Build DPO pairs + COMET filtering dpo/prepare_data/ 1 × 1 1 h limit
7 DPO training dpo/training/ 2 × 4 4 h limit
8 Merge LoRA adapter merge_lora/ 1 × 1 30 min limit
9 GRPO training grpo/ 3 × 4 4 days limit
10 Merge LoRA adapter merge_lora/ 1 × 1 30 min limit

The merge step appears three times because every stage trains a LoRA adapter and every stage expects a complete standalone model as input. It is described once, in its own chapter.

What every sbatch script needs from you

Every batch script in the repository begins with an empty WORK_DIR that must be filled in with the absolute path of your clone:

# TODO: Add path to the root dir of the GaMS-Translator repository
WORK_DIR=/leonardo_work/<your_account>/GaMS-Translator

Everything else (container path, model directory, data directory, log locations, container bind mounts) is derived from it. Scripts that log to WandB have a second WANDB_API_KEY TODO, and download_models.sbatch has a third for HF_TOKEN. Each stage chapter repeats the TODOs relevant to it, so you can work through the pipeline chapter by chapter without hunting.

The three training scripts (sft/run_sft.sbatch, dpo/training/run_dpo.sbatch and grpo/run_grpo.sbatch) follow the same approach. They do not run Python directly under srun. Instead, they generate a shell script into the stage's experiments/ directory, copy themselves next to it, and run the generated script inside the container. The effect is that every run leaves behind a complete record of the exact commands and the exact batch script that produced it, which is what makes an experiment from three months ago reproducible. The other four scripts (the download job, the translation job, the DPO preparation job and the merge job) are ordinary batch scripts that invoke hf or python3 directly and generate nothing.

Hyperparameters at a glance

The stages differ in more than their objective. Collecting the numbers in one place makes the shape of the pipeline visible:

SFT DPO GRPO
Examples ~900 train / 112 val ~1,500 ~103 k train / 1,018 val
Epochs 3 3 3
Global batch size 4 8 64 prompts
Learning rate 2e-5 5e-6 5e-6
Minimum LR 1e-6 1e-6 5e-7
Warmup steps 200 50 1000
KL beta - 0.1 0.04
Max length 8192 (prompt + completion) 8192 (prompt + completion) 4096 (prompt) + 4096 (completion)
LoRA rank 128 32 128
Liger kernel yes no no
Nodes × GPUs 1 × 4 2 × 4 3 × 4

DPO's and GRPO's learning rates are an order of magnitude below SFT's. This is standard for preference tuning. SFT is teaching the model a task it does not know, while DPO and GRPO are steering a model that already performs the task. A learning rate appropriate for SFT would destroy the model during preference-optimization.

A note on Liger

use_liger_kernel=True appears in the SFT configuration and is absent from the DPO and GRPO configurations. In SFT, use_liger_kernel patches operations inside the model's own forward pass, which remains compatible with DeepSpeed's just-in-time parameter gathering under ZeRO-3. In DPO and GRPO, TRL instead swaps in a fused-linear loss that reaches into the model, pulls lm_head.weight out directly and hands it to the kernel - with no deepspeed.zero.GatheredParameters guard around it. Under ZeRO Stage 3 that weight is an unmaterialized shard at that moment, and training fails. If you enable Liger for a preference stage and get an inscrutable error about an empty or zero-sized tensor in the loss, this is why.