Preference Optimization Pipeline and LLM-Based Machine Translator for English and Slovene Pairs

GaMS GRPO translators and the preference optimization pipeline

Opis

This service provide trained machine translation models specialized for English to Slovene translation. It shows how to run these models locally or on an HPC cluster. It also provides a pipeline for training machine translators with preference optimization. This pipeline can also be applied for other tasks, where the response quality can be automatically evaluated. It is based on Slovene LLM GaMS.

Pričakovani rezultati

Following the resources provided in this service, user will be able to: (1) Use the published GaMS GRPO translators for English to Slovene translation with the correct instruction prompt; (2) Run large scale offline batch translation of a document collection using vLLM; (3) Deploy a translator as a persistent OpenAI compatible inference server with a web interface; (4) Perform preference-based optimization of machine translators; (5) Generate on-policy model outputs and build preference pairs from the model's own systematic errors; (6) Write reward functions and perform GRPO training with a dedicated vLLM generation server

Kartica storitve

Kategorija	AI development and deployment
Domena	Natural Language Processing, Machine Translation, Preference Alignment
Vertikala	Cross-domain
Ciljni uporabniki	Industry, Large Enterprises, Scientific Institutions, Public Institutions
Tip storitve	Model, Pipeline
Rezultat	Preference-tuned machine translator for English and Slovene.
Zahteve	Requires several NVIDIA GPUs.
Omejitve
Preizkušene platforme	HPC LEONARDO, HPC Vega
Na voljo od	Q3/2026
Licenca	MIT
Slika
Oznaka slike
Repozitorij	GaMS-Translator
Modeli	GaMS3-12B-GRPO-Translator, GaMS2-27B-GRPO-Translator

Introduction

Machine translation is one of the oldest tasks in natural language processing, and for a long time it was also one of the most specialised: dedicated encoder-decoder architectures, trained on sentence-aligned parallel corpora, dominated the field. Large language models changed that. A modern LLM translates by being told to translate, and because it reads the whole document rather than one sentence at a time, it can keep pronouns consistent, resolve ambiguous terminology from context, and preserve the structure of the text it was given.

The catch is that an instruction-tuned general-purpose LLM is not a translator. Asked to translate a document, it will produce something reasonable and then keep going, or add a helpful comment, or quietly collapse the paragraph structure. These are the failures of behaviour as instruction-tuned models are trained to be helpful general assistants instead of specialised translators. Hence, it makes sense to specialise the pretrained LLM, that is fluent in target language (such as GaMS for Slovene), to perform the translation task. Supervised fine-tuning can be problematic for translation task, as it requires a large enough high-quality dataset. However, for an English to Slovene translation we do not have such documents at scale.

Preference optimization solves this by changing what the training signal is. Instead of "here is the correct answer, reproduce it", the signal becomes "of these candidate answers, this one is better than that one". The crucial part is that all of the candidates are produced by the model itself. This is the family of methods that turned base language models into usable assistants, and it is what this service is about. Machine translation serves as the working example throughout, for two reasons: the failure modes are concrete and easy to see, and translation quality can be scored automatically by a neural metric, which means the reward signal can be produced without a human in the loop.

The service consists of three parts:

1. Two ready-to-use translation models, published on the HuggingFace portal under the CJVT organization: cjvt/GaMS3-12B-GRPO-Translator and cjvt/GaMS2-27B-GRPO-Translator. These are the output of the pipeline described in this book, and for the majority of readers they are the whole service - they can be deployed today, without a single GPU-hour of training.
2. A GitHub repository containing the complete three-stage training pipeline (SFT → DPO → GRPO) that produced those models, written for a SLURM-based EuroHPC cluster.
3. This book, which explains how to use the published models and how the pipeline that produced them works, so that the same recipe can be applied to a different task, language pair, or reward function.

Book content

The book is in two parts.

Part 1 - the trained translators. We start with what the models are and what to expect from them. Then we cover how to run them using three different deployments. The first is using the transformers library for trying the model out. The second is large-scale offline batch inference with vLLM, for translating a corpus. Here we lean heavily on the PEFT service, which covers the machinery in depth, and concentrate on what is specific to document-level translation. The third is a persistent OpenAI-compatible inference server with a web interface, which is how you make the translator available to your colleagues or other applications.

Part 2 - the training pipeline. Here we work through the three consecutive stages that turn a pretrained GaMS model into a translator. Supervised fine-tuning first teaches the model the chat format and the shape of the task. DPO comes second and trains the model to avoid specific formatting mistakes through on-policy training. Finally, during the GRPO stage the model generates groups of candidate translations, four reward functions score them, and the policy is updated toward the above-average candidates. We explain each reward function, the three-node trainer/inference-server topology that makes online generation practical, and the failure modes worth watching for. A closing chapter discusses what has to change to use the pipeline for a different task.

Read the first part first

Preference optimization is expensive. The GRPO stage that produced these models occupied two DGX B200 nodes for several days. Every optimizer step waits for the model to generate fresh candidate translations and for a neural metric to score all of them. That cost is unavoidable and inherent to online reinforcement learning, but it is also a cost that has already been paid. If your task is English → Slovene translation, or something close enough to it, download the model and skip Part 2 entirely.

Part 2 earns its place when the published models are not what you need: a different language pair, a different domain (in which generic translation quality is not the right objective), a different task altogether (that can be automatically scored). In that case the pipeline, the reward functions and the cluster configuration in this book are a working starting point. If you need to adapt the model for your specific need, start with SFT and then continue with DPO as they are way cheaper than GRPO. If the model still does not achieve the desired results, only then use GRPO.

Relationship to the PEFT fine-tuning service

This service build on the SLAIF service Adaptation of a Large Language Model for a Specific Task using Parameter Efficient Fine-Tuning. Everything that is common to any LoRA-based LLM training run on an HPC cluster is explained in detail there. This includes building an Apptainer/Singularity container, what LoRA is and how LoraConfig is set, how DeepSpeed ZeRO-3 shards a model that does not fit on one GPU, Flash Attention 2, Liger kernels, gradient checkpointing, the learning-rate schedule, the global/micro batch size distinction, WandB tracking, checkpoint resumption, merging an adapter back into the base model, and large-scale offline batch inference with vLLM.

We do not repeat any of it here. Where a step in this pipeline is the same as in the PEFT service, this book says so, points at the corresponding chapter of the PEFT service, and describes only what is different. The chapters that follow therefore assume you have either read that service book or are comfortable with its material. In this book we focus on the preference optimization part.

GaMS

GaMS (Generative Model for Slovene) is a family of large language models developed within the PoVeJMo project. The models are open-source, specialized for Slovene, and their weights are openly available on the HuggingFace portal under the CJVT organization. They are based on Google's Gemma and come in several versions, differing in parameter count, starting model (Gemma 2 for GaMS2, Gemma 3 for GaMS3) and training stage. Instruct versions are instruction-tuned and ready for prompt-completion use, while versions without Instruct in the name are pretrained versions.

Two properties make GaMS a good fit for this service in particular. First, the models genuinely know Slovene (its morphology, its word order, its register) which is the half of the translation problem that cannot be fixed by training on formatting. Second, at 12 to 27 billion parameters they are large enough to translate well and small enough that a preference-optimization run over them is feasible.

Part 1: The Trained Translators

Two English → Slovene translation models are published as part of this service:

Model	Base model	Parameters	Generation
cjvt/GaMS3-12B-GRPO-Translator	cjvt/GaMS3-12B	12 B	Gemma 3
cjvt/GaMS2-27B-GRPO-Translator	cjvt/GaMS2-27B	27 B	Gemma 2

Both are released under the Gemma terms of use, inherited from the Gemma models their GaMS bases were built from. Note that this is the licence of the model weights and is separate from the Apache 2.0 licence of the pipeline code - if you plan to deploy either model commercially, read the Gemma terms rather than assuming the code licence applies.

How they were built

Each model is the result of the same three consecutive training stages, and each stage trained a LoRA adapter that was merged into the model before the next stage began:

GaMS base ──SFT──> merge ──> SFT-Translator
                                  │
                                  └──DPO──> merge ──> DPO-Translator
                                                            │
                                                            └──GRPO──> merge ──> GRPO-Translator

1. SFT on roughly 1,000 high-quality English → Slovene translation examples. This stage is small on purpose: its job is to teach the pretrained model the chat template and the shape of the task, not to teach it to translate.
2. DPO on approximately 1,500 on-policy preference pairs - pairs constructed from the SFT model's own translations, targeting the specific formatting failures it exhibited: collapsed paragraph separators, failing to stop at the end of the document, and dates left in English format.
3. GRPO on roughly 100,000 examples, with four reward functions (COMET quality, language identification, length ratio and BLEU against a reference). This is the main training stage and the one the models are named after.

Part 2 of this book describes all three stages in full, including the code that ran them.

Which model to choose

The 12 B model is the default recommendation. It is based on the newer Gemma 3 generation, fits comfortably on a single 40 GB GPU in bfloat16, and is roughly half the memory and a good deal faster than the 27 B model at inference time. The 27 B model is worth evaluating if you have the hardware and translation quality is the binding constraint rather than throughput; larger models generally produce higher-quality translations.

Both are document-level translators. They were trained on whole documents - Wikipedia articles, news articles, web text - rather than isolated sentences. As a result they should perform more consistent across the document than sentence-level based translators.

Attention backend for the 27 B model

The GaMS2-27B-GRPO-Translator model card reports observed quality degradation when the model is served with the FlashInfer or ROCm attention backends. Use FlashAttention 2 (on most devices) or FlashAttention 3 (on Hopper devices). With vLLM this is controlled by the VLLM_ATTENTION_BACKEND environment variable; check which backend was selected in the server's start-up log before drawing conclusions about output quality.

The prompt is part of the model

Both models were trained with the same prompt through all three stages. Using a different instruction, translating the instruction into English, or dropping it entirely puts the model outside the distribution it was optimized on. It might still work outside this distribution, but its behavior will be less predictable.

The prompt has three parts: - a role description - a task instruction, - an English text to translate. The three parts are separated by a blank line (double newline):

Si profesionalni prevajalec iz angleščine v slovenščino. Tvoja naloga je ustvariti visoko kakovostne prevode, ki so naravni, tekoči in natančni. Izogibaj se dobesednim prevodom, ki zvenijo nenaravno. Upoštevaj kontekst, idiome in kulturne reference. Ne dodajaj nobenih dodatnih informacij ali komentarjev, samo prevod.

Prevedi podano angleško besedilo v slovenščino.

<English text goes here>

In Python it is convenient to keep the instruction in one constant and interpolate the document, which is exactly what the pipeline code does - the constant below is copied verbatim from PROMPT_TEMPLATE in grpo/reward_functions.py, and the same string appears in dpo/get_translations/translate.py and dpo/prepare_data/select_training_data.py:

PROMPT_TEMPLATE = """\
Si profesionalni prevajalec iz angleščine v slovenščino. Tvoja naloga je ustvariti visoko kakovostne prevode, ki so naravni, tekoči in natančni. Izogibaj se dobesednim prevodom, ki zvenijo nenaravno. Upoštevaj kontekst, idiome in kulturne reference. Ne dodajaj nobenih dodatnih informacij ali komentarjev, samo prevod.

Prevedi podano angleško besedilo v slovenščino.\
"""

english_text = "This is the text to translate."
message = [{"role": "user", "content": f"{PROMPT_TEMPLATE}\n\n{english_text}"}]

Three details in that snippet are easy to get wrong and worth stating explicitly:

• The instruction and the document go in the same user turn, separated by exactly one blank line (\n\n). The instruction is not a system message. GaMS3 and Gemma 3 chat templates do accept a system role, but convert it internally to exactly the same format as it is seen above. Gemma 2 based GaMS2 does not support system role at all.
• There is exactly one turn. The models are used single-turn: one user message in, one assistant message out. They are not conversational translators and were never trained to carry a translation across multiple turns.
• The trailing backslash in the Python string matters. """\ at the start and \ before the closing quotes keep the constant free of a leading and trailing newline, so that the \n\n you add yourself is the only separator. If you retype the prompt by hand and leave a trailing newline in it, the model sees three newlines instead of two.

Both models use the same chat template as their Instruct siblings (cjvt/GaMS3-12B-Instruct and cjvt/GaMS2-27B-Instruct) and the templates are part of their repositories.

What to expect, and what not to

The models translate English into Slovene. They are not general assistants anymore. The training pipeline pushed the policy hard toward a single behavior, and asking one of these models a question instead of giving it a document to translate is not a supported use. If you need both capabilities, serve the corresponding Instruct models instead.

Translation direction is English → Slovene. The reverse direction was not trained and not evaluated. It might work anyway, but is not officially supported.

Structure is preserved rather than reformatted. The training data used Markdown-flavoured documents - a # Title first line, paragraphs separated by blank lines, and for news articles an italic *D. M. YYYY* date line under the title. The DPO stage specifically taught the model to keep the paragraph structure of the source and to convert English *YYYY-MM-DD* dates into Slovene format. If your input follows that convention, the output will too. If your input is a wall of unstructured text, expect a wall of unstructured Slovene.

Running the Translators

There are many different ways to run these models. In this book we present three:

Deployment	Use it when	Covered in
transformers pipeline	Trying the model out, translating a handful of documents, prototyping	Model cards
vLLM offline batch inference	Translating a corpus: thousands of documents, one job, no interactivity	PEFT service
vLLM OpenAI-compatible server	The translator is a service other people or applications call	SLAIF Workshop

The starting point: transformers

The easiest way to see the model work is the transformers pipeline API. This loads the full model onto the device and generates one response at a time, which is slow and memory-hungry compared to vLLM but requires nothing beyond transformers, torch and accelerate (latest is required only for multi-GPU inference). The code for using the translators with transformers is part of the translators' model cards:

from transformers import pipeline

model_id = "cjvt/GaMS3-12B-GRPO-Translator"
model = pipeline(
    "text-generation",
    model=model_id,
    device_map="cuda"  # replace with "mps" to run on a Mac device
)

prompt_template = "Si profesionalni prevajalec iz angleščine v slovenščino. Tvoja naloga je ustvariti visoko kakovostne prevode, ki so naravni, tekoči in natančni. Izogibaj se dobesednim prevodom, ki zvenijo nenaravno. Upoštevaj kontekst, idiome in kulturne reference. Ne dodajaj nobenih dodatnih informacij ali komentarjev, samo prevod.\n\nPrevedi podano angleško besedilo v slovenščino."
english_text = "This is the text to translate."

message = [{"role": "user", "content": f"{prompt_template}\n\n{english_text}"}]
response = model(message, max_new_tokens=512)

print("Translation:", response[0]["generated_text"][-1]["content"])

Two things to keep in mind. First, max_new_tokens=512 is the example value, not a sensible default for document translation — a Wikipedia article will be truncated mid-sentence at 512 tokens. Size it against your longest input; Slovene output is typically a little longer in tokens than the English source, so a factor of roughly 1.5× the source token count is a reasonable starting budget. Second, device_map="cuda" puts the whole model on one GPU. The 12 B model in bfloat16 needs about 24 GB of weights plus the KV cache, so it fits on a 40 GB card; the 27 B model needs about 54 GB and will not, so use device_map="auto" to shard it across cards or serve it with vLLM instead.

Batch inference at scale

A simple transformers call is too slow for translating the corpus. For large-scale translations we need to combine much faster vLLM library (there are other libraries optimized for inference as well) with the batch inference approach. Moreover, it makes sense to distribute translation tasks across multiple GPUs, each translating a different part of the corpus. Batch inference is described in-detail in the Batch Inference at Scale with vLLM chapter of PEFT service and we do not repeat it here. That chapter covers:

• PagedAttention and why looping over model.generate() is an order of magnitude slower than batched vLLM inference,
• tensor parallelism via tensor_parallel_size, and setting it from $SLURM_GPUS_ON_NODE so the model is split across every GPU on the node,
• SLURM array jobs and interleaved dataset sharding, where --array=0-7 launches eight independent jobs and each takes dataset[shard_idx::n_shards],
• incremental writing with flush() after every batch so an interrupted job keeps its completed work,
• and merging the per-shard JSONL outputs afterwards,

Sampling parameters for translation

The PEFT service uses a temperature of 0.6 with top_p=0.9, which is a good general-purpose setting for open-ended generation. Translation is not open-ended generation, and the recommended settings here are different.

The model cards suggest temperature=0.2 with top_p=0.9. The reasoning is that translation is a task, where the model does not need to produce a new content, but instead map the existing content to a new language. As we do not need diverse answers, the sampling diversity would only hurt the accuracy. A low temperature sharpens the distribution toward the model's highest-confidence continuation. This is also consistent with how the models were trained: the GRPO stage used temperature=0.1.

Greedy decoding (temperature=0) is an option too. It is also good for reproducibility If you need identical output across runs, use it.

Setting Up an Inference Server

Offline batch inference is the right tool when you have a corpus and a deadline. It is the wrong tool when the translator has to be available - when you want to serve the translator through UI where user can paste a document into a text box, or another application wants to call an API. For that you want a persistent server.

The setup described here is taken from the DSI-SLAIF workshop, whose third exercise (03_local_server/) builds exactly this stack for a GaMS model. That repository is the reference implementation and contains the complete, runnable configuration; this chapter explains what the pieces do and what to change to serve a translator instead of a general-purpose Instruct model.

Three components, three ports

The stack is three containers orchestrated with Docker Compose:

   browser                  application code
      │                            │
      ▼                            ▼
┌─────────────┐            ┌────────────────┐
│  OpenWebUI  │ ─────────► │ LiteLLM proxy  │
│   :3000     │            │     :4000      │
└─────────────┘            └────────────────┘
                                   │
                                   ▼
                          ┌────────────────┐
                          │  vLLM server   │
                          │     :8001      │  ← GPU
                          └────────────────┘

vLLM does the actual work. The same engine used for offline batch inference in the previous chapter also has a server mode which exposes an OpenAI-compatible HTTP API - the same /v1/chat/completions endpoint shape as the commercial APIs, which means any OpenAI client library can talk to it. This is the component that holds the model on the GPU.

LiteLLM sits in front of vLLM as an API gateway. We could pass the vLLM server directly to the OpenWebUI, but LiteLLM has its advantages. It gives you a single place to issue and revoke API keys (each vLLM server supports only one API key), to log requests, and to add rate limits. This means that you can create different API keys for different users and set custom rate limits for each API key. Moreover, LiteLLM lets you put several models (and backends) behind one endpoint: a translator and a general-purpose Instruct model can be two entries in one configuration file, served from separate vLLM instances (or another backend, for example Ollama) but selected by the client through the model field of a single API.

OpenWebUI is a ChatGPT-like web frontend. It is configured to treat LiteLLM as its OpenAI backend, so anything LiteLLM serves shows up in the model dropdown.

The vLLM service

The workshop drives vLLM through a config file rather than a long command line, which keeps the serving parameters in version control:

# GaMS3_server/config/server_config.yaml -- as shipped by the workshop
port: 8001
api-key: "GaMS-server-key"
gpu-memory-utilization: 0.4
override-generation-config: '{"temperature": 0.6, "top_p": 0.9}'

Every key in that file is the long form of a vllm serve command-line flag, and the server is started by pointing at both the model and the config:

# docker-compose.yml (vllm service)
entrypoint: ["vllm", "serve", "/models/GaMS3-12B-Instruct",
             "--config", "/config/server_config.yaml"]

The workshop serves GaMS3-12B-Instruct, since its subject is a general-purpose assistant. To serve a translator instead, change that model path. Do not forget to change it in the LiteLLM configuration too, since the two files have to agree.

The following settings deserve further explanation:

• gpu-memory-utilization is the fraction of GPU memory vLLM is allowed to claim for weights plus KV cache. The workshop sets 0.4 because the exercise was run on DGX Spark. Adjust it for your hardware.
• override-generation-config sets server-side defaults for sampling. The workshop's value of 0.6 is tuned for a general-purpose Instruct model. For a translator, lower it to the recommended temperature=0.2 / top_p=0.9 from the previous chapter.
• port must match what LiteLLM is configured to call. The workshop uses 8001 rather than vLLM's default 8000 because port 8000 was already occupied during the workshop.

The model is served through local files, so it does not have to be downloaded at each vLLM server restart. The models dir is mounted through MODELS_DIR variable, that is set in .env file. Set the variable path to a dir, where you pre-downloaded the translators using hf download.

The LiteLLM proxy

LiteLLM's configuration is a list of model entries, each mapping a public name to a backend:

# litellm/litellm_config.yaml -- translator substituted for the workshop's Instruct model
model_list:
  - model_name: GaMS3-12B-GRPO-Translator
    litellm_params:
      model: openai//models/GaMS3-12B-GRPO-Translator
      api_base: http://vllm:8001/v1
      api_key: GaMS-server-key

model_name is what clients ask for. model is a combination of provider backend (openai/ here, because vLLM speaks the OpenAI protocol) and model identifier (model path given to the vLLM). api_base uses the Compose service name vllm as the hostname, which resolves inside the Compose network. api_key is the secret from server_config.yaml.

Adding a second model is another entry in the same list. This is how you serve a translator and a chat model side by side:

model_list:
  - model_name: GaMS3-12B-GRPO-Translator
    litellm_params:
      model: openai//models/GaMS3-12B-GRPO-Translator
      api_base: http://vllm:8001/v1
      api_key: GaMS-server-key
  - model_name: GaMS3-12B-Instruct
    litellm_params:
      model: openai//models/GaMS3-12B-Instruct
      api_base: http://vllm-instruct:8002/v1
      api_key: GaMS-server-key

Each model is served through a separate vLLM container with its own GPU allocation.

The prompt is still your responsibility

Neither vLLM nor LiteLLM can add the translation instruction for you, and a request that omits it will be answered - badly. If the server is going to be called by people or code that should not have to know about a Slovene instruction prefix, wrap it: a thin FastAPI service that accepts {"text": ...}, prepends PROMPT_TEMPLATE, forwards to LiteLLM and returns the translation is a few dozen lines, and it puts the one detail that must not be got wrong in exactly one place.

Where OpenWebUI fits

OpenWebUI is a full chat interface, which makes it a slightly odd fit for a single-turn translator: the model will happily translate the first message and then, on the second, translate whatever it is given in the context of a conversation it was never trained to have. It is nonetheless the fastest way to let someone evaluate translation quality without writing code, and its system-prompt feature can be used to keep the instruction in place across a session.

For a translator specifically, a purpose-built page (a source box, a target box, and a button) is often the better answer, and with an OpenAI-compatible endpoint already in place it is a small amount of frontend work.

Bringing the stack up

From the directory containing docker-compose.yml:

docker compose up -d
docker compose logs -f

Loading a 12 B model takes a couple of minutes; watch the vLLM logs until it reports that the API server is running. OpenWebUI is then reachable at http://localhost:3000 and, with WEBUI_AUTH=False as the workshop sets it, needs no login - convenient for a workshop, not appropriate for a shared deployment, so turn authentication on if the interface is reachable by anyone but you.

To stop everything:

docker compose down

Part 2: The Training Pipeline

Everything from here on describes how the models in Part 1 were made. If you can use those models as they are, you do not need this part. Read on if you want a translator for a different language pair, a translator tuned to a domain where generic quality not enough, or a model preference-optimized for some entirely different task for which you can write down a scoring function.

The code lives in the GaMS-Team/GaMS-Translator repository and is written for the LEONARDO EuroHPC supercomputer (SLURM + Singularity). It runs on other SLURM clusters (given the strong enough hardware) with minor changes, mainly to partition names and time limits. Where LEONARDO's specifics matter, this book says so.

Why three stages

The pipeline is three consecutive training stages, and each one exists because the previous one cannot do its job:

GaMS3-12B ──SFT──> merge ──> GaMS3-12B-SFT-Translator
                                      │
                                      ├──> generate translations ──> DPO pairs
                                      │
                                      └──DPO──> merge ──> GaMS3-12B-DPO-Translator
                                                                    │
                                                                    └──GRPO──> merge ──> GaMS3-12B-GRPO-Translator

SFT teaches the task and the format. A pretrained GaMS model does not know the chat template and has no notion that a user turn containing a Slovene instruction and an English document should be answered with a Slovene translation and nothing else. A thousand high-quality examples fix that.

DPO fixes the specific things the SFT model gets wrong. After SFT the model translates competently and formats badly: it collapses paragraph separators, it keeps generating past the end of the document, it leaves dates in English form. These are systematic, identifiable errors, which makes them the ideal target for preference learning. For each of them we can take the model's own output and repair it programmatically, producing a pair where the difference between chosen and rejected is exactly the behaviour we want to change. This is what on-policy means in practice, and it is the reason the DPO stage needs only about 1,500 pairs to be effective. If you observe an error in published models that can be systematically corrected, you can quickly fix it with DPO stage.

GRPO improves translation quality itself. DPO learns from a fixed set of pairs, which clearly show the difference between the desired and bad response. Translation quality is not like that. It is difficult to turn a mediocre translation into a good one systematically. Instead, we can estimate the quality of each translation through reference-free metrics such as CometKiwi. Using GRPO, we can ask the model to translate the same example multiple times, score each translation and turn scores into a gradient through a reward function. GRPO pushes the model toward the candidates (translations) with higher scores than their group's average. Unlike DPO, the samples are drawn fresh from the current policy at every step, which is why this stage is the expensive one.

Stage map

#	Stage	Directory	Nodes × GPUs	Wall time
1	Build the Singularity container	singularity/	login node	minutes
2	Download models, reward models and GRPO data	-	1 (serial, no GPU)	up to 2 h
3	SFT training	sft/	1 × 4	8 h limit
4	Merge LoRA adapter	merge_lora/	1 × 1	30 min limit
5	Generate on-policy translations for DPO	dpo/get_translations/	1 × 4	2 h limit
6	Build DPO pairs + COMET filtering	dpo/prepare_data/	1 × 1	1 h limit
7	DPO training	dpo/training/	2 × 4	4 h limit
8	Merge LoRA adapter	merge_lora/	1 × 1	30 min limit
9	GRPO training	grpo/	3 × 4	4 days limit
10	Merge LoRA adapter	merge_lora/	1 × 1	30 min limit

The merge step appears three times because every stage trains a LoRA adapter and every stage expects a complete standalone model as input. It is described once, in its own chapter.

What every sbatch script needs from you

Every batch script in the repository begins with an empty WORK_DIR that must be filled in with the absolute path of your clone:

# TODO: Add path to the root dir of the GaMS-Translator repository
WORK_DIR=/leonardo_work/<your_account>/GaMS-Translator

Everything else (container path, model directory, data directory, log locations, container bind mounts) is derived from it. Scripts that log to WandB have a second WANDB_API_KEY TODO, and download_models.sbatch has a third for HF_TOKEN. Each stage chapter repeats the TODOs relevant to it, so you can work through the pipeline chapter by chapter without hunting.

The three training scripts (sft/run_sft.sbatch, dpo/training/run_dpo.sbatch and grpo/run_grpo.sbatch) follow the same approach. They do not run Python directly under srun. Instead, they generate a shell script into the stage's experiments/ directory, copy themselves next to it, and run the generated script inside the container. The effect is that every run leaves behind a complete record of the exact commands and the exact batch script that produced it, which is what makes an experiment from three months ago reproducible. The other four scripts (the download job, the translation job, the DPO preparation job and the merge job) are ordinary batch scripts that invoke hf or python3 directly and generate nothing.

Hyperparameters at a glance

The stages differ in more than their objective. Collecting the numbers in one place makes the shape of the pipeline visible:

	SFT	DPO	GRPO
Examples	~900 train / 112 val	~1,500	~103 k train / 1,018 val
Epochs	3	3	3
Global batch size	4	8	64 prompts
Learning rate	2e-5	5e-6	5e-6
Minimum LR	1e-6	1e-6	5e-7
Warmup steps	200	50	1000
KL beta	-	0.1	0.04
Max length	8192 (prompt + completion)	8192 (prompt + completion)	4096 (prompt) + 4096 (completion)
LoRA rank	128	32	128
Liger kernel	yes	no	no
Nodes × GPUs	1 × 4	2 × 4	3 × 4

DPO's and GRPO's learning rates are an order of magnitude below SFT's. This is standard for preference tuning. SFT is teaching the model a task it does not know, while DPO and GRPO are steering a model that already performs the task. A learning rate appropriate for SFT would destroy the model during preference-optimization.

A note on Liger

use_liger_kernel=True appears in the SFT configuration and is absent from the DPO and GRPO configurations. In SFT, use_liger_kernel patches operations inside the model's own forward pass, which remains compatible with DeepSpeed's just-in-time parameter gathering under ZeRO-3. In DPO and GRPO, TRL instead swaps in a fused-linear loss that reaches into the model, pulls lm_head.weight out directly and hands it to the kernel - with no deepspeed.zero.GatheredParameters guard around it. Under ZeRO Stage 3 that weight is an unmaterialized shard at that moment, and training fails. If you enable Liger for a preference stage and get an inscrutable error about an empty or zero-sized tensor in the loss, this is why.

Environment Setup and Downloads

Cloning and building the container

git clone https://github.com/GaMS-Team/GaMS-Translator.git
cd GaMS-Translator

All paths in this part of the book are relative to this repository root.

As in the PEFT service, the entire software environment is packed into a single container image, so that a job behaves identically on one node or twelve and two years from now on a different cluster. The recipe is at singularity/recipe.def and the image is built into the same directory:

cd singularity/
singularity build gams_translator.sif recipe.def
cd ..

The command is singularity rather than apptainer because that is what LEONARDO provides. It uses the same recipe format as Apptainer (Apptainer is the modern fork of Singularity). Everything the PEFT service says about container builds applies here, including the --fakeroot situation. See the Environment Setup chapter of the PEFT service for the full discussion.

The software stack

The container starts from nvidia/cuda:12.8.1-cudnn-devel-ubuntu24.04 and installs everything into a virtual environment at /opt/venv:

Library	Version
PyTorch	2.8.0
Transformers	4.56.1
TRL	0.25.1
DeepSpeed	0.18.1
vLLM	0.10.2
FlashAttention	2.8.3 (prebuilt wheel for cu128 / torch 2.8)
WandB	0.25.1
PEFT, Accelerate	latest
Liger Kernel	latest
unbabel-comet, fasttext, sacrebleu, loguru	latest

The vLLM version matters more here than it does for pure inference. In the GRPO stage vLLM is not just an inference engine we use after training. It is a live server that TRL pushes updated weights into on every step, through the trl vllm-serve command. That integration is version-sensitive on both sides, so the TRL and vLLM versions in this recipe are a validated pair. If you upgrade one, upgrade the other and expect to test.

Compared to the PEFT service, we added the libraries that compute the reward functions. unbabel-comet provides the COMET quality metric, fasttext the language identifier, and sacrebleu the BLEU implementation.

Downloading models and data

Since LEONARDO's GPU partition does not have internet access, we need to download the models and data before training. download_models.sbatch fetches everything the pipeline needs in one job:

Repository	Destination	Purpose
cjvt/GaMS3-12B	models/GaMS3-12B	Base model for SFT
cjvt/GaMS3-12B-Instruct (tokenizer files only)	models/GaMS3-12B-Instruct	Tokenizer (chat template)
Unbabel/wmt22-cometkiwi-da	translation_reward_models/wmt22-cometkiwi-da	COMET scoring
fastText lid.176.bin	translation_reward_models/lid.176.bin	Language identification
cjvt/GaMS-Translator-GRPO-Training	data/grpo_training	GRPO training data

We need to download the training data only for the GRPO stage. The SFT data ships with the repository in data/sft_training/, and the DPO data is generated by the pipeline itself.

SFT starts from the pretrained GaMS3-12B, which has no chat template (similar to PEFT service). To get the chat template we need to download GaMS3-12B-Instruct repository. However, we do not need the whole Instruct model, meaning that downloading weights would result in downloading and storing additional 24 GB of weights we do not need. Hence, we add the --include flag to the standard hf download command to make sure that only tokenizer files are downloaded:

hf download cjvt/GaMS3-12B-Instruct \
  --local-dir /models/GaMS3-12B-Instruct \
  --include "tokenizer*" "special_tokens_map.json" "added_tokens.json" "chat_template*"

The COMET model is gated. Unbabel/wmt22-cometkiwi-da requires you to accept its licence at its model page while logged in. To authenticate yourself, you need to provide your HF access token to the download script:

# TODO: Enter your HuggingFace access token.
export HF_TOKEN=

Without it the download fails with a 401. The download job does not stop on that failure, it simply logs it and carries on to the next step, so check the log, and check the models dir after the download. Confirm that file exists before submitting anything downstream.

The GRPO data is a dataset repository, not a model, so it is fetched with --repo-type dataset. It is public and needs no token, but it is licensed CC BY-NC 4.0 (a non-commercial clause), which is stricter than the pipeline code's own licence. If you intend to train a commercial model, this dataset is not usable as-is and you will need to construct your own GRPO data. The non-commercial license is used as some ground-truth translations (used for BLEU score) were translated with Gemini. The download is about 406 MB. Because the dataset repository keeps its parquet shards in a data/ subdirectory, they land at data/grpo_training/data/. grpo/run_grpo.sbatch binds that inner directory as /data.

Running the download script

sbatch download_models.sbatch

Progress is logged in $WORK_DIR/download_logs.txt.

hf versus huggingface-cli

The script uses the modern hf download CLI, which requires huggingface_hub >= 0.34. On older containers the command is huggingface-cli download instead. A comment in the script marks the lines to change.

Stage 1: Supervised Fine-Tuning

The SFT stage is mechanically identical to the one the PEFT service describes in detail. Same SFTTrainer, same prompt/completion data format with completion_only_loss=True, same LoRA configuration, same DeepSpeed ZeRO-3 setup, same cosine-with-minimum learning-rate schedule, same WandB integration, same checkpoint-resume mechanism. For what the rank means, why alpha is twice the rank, why all seven projections rather than just query and value, and the arithmetic of how much memory this saves, read the Supervised Fine-tuning using LoRA chapter of the PEFT service. The same chapter covers DeepSpeed ZeRO-3, Flash Attention 2, gradient checkpointing and the global/micro batch size distinction, all of which are used here unchanged.

What is the purpose of this stage

Model does not learn to translate during the SFT stage. GaMS already knows English and Slovene, having seen enormous quantities of both during pretraining. The SFT stage shows the model how to connect this existing knowledge into a translation. It teaches the model that a user turn shaped like "instruction, blank line, English document" is answered with a Slovene translation of that document, ending where the document ends, with no preamble and no commentary. The consequence is that the stage is cheap and its output is not the final deliverable. GaMS3-12B-SFT-Translator is an intermediate artifact, and its main job is to produce the raw material for the next two stages.

The PEFT service discusses at length when to start from a pretrained model versus an Instruct model, and notes there that preference alignment requires a model which already understands the conversation format. We start from pretrained cjvt/GaMS3-12B rather than cjvt/GaMS3-12B-Instruct, and we use SFT to install the format ourselves. The reason is that the Instruct model was tuned to be a helpful general-purpose assistant, and helpfulness is a liability for a translator: an assistant explains, hedges, adds context, and asks whether you would like anything else. All of that has to be trained out again. Starting from the pretrained model and teaching it exactly one behaviour is cleaner, and it is affordable because the behaviour is simple.

Input data

SFT reads two JSONL files from data/sft_training/, which are mounted into the container as /data. The format is the standard TRL prompt/completion format:

{
  "corpus": "fineweb",
  "prompt":     [{"role": "user",      "content": "<translation instruction + English document>"}],
  "completion": [{"role": "assistant", "content": "<Slovene translation>"}]
}

The prompt field already contains the full instruction: the PROMPT_TEMPLATE from Part 1, two newlines, then the English document.

Configuration

The parameters that differ from the PEFT service's SFT run:

• MAX_LENGTH = 8192 rather than 16,384. This length ensures that medium-length documents can be translated (it supports documents with up to approximately 3,000 words) and that the training remains feasible during the later stages (both in terms of vRAM and compute time). To translate longer documents, you should first split the document into smaller chunks.
• micro_batch_size = 1, global batch_size = 4. Small, because the dataset is small: with 891 examples a global batch of 4 gives about 222 optimizer steps in the single epoch, which is enough to learn a format. A larger global batch would give fewer steps and underfit. Note: with a global batch of 4 you cannot use more than 4 GPUs for this stage without the accumulation count going to zero.
• 3 epochs, with evaluation and checkpointing twice per epoch. We do not need to evaluate as often as in PEFT Service, as we have significantly smaller training data.

Before running

Open sft/run_sft.sbatch and fill in:

WORK_DIR=/path/to/GaMS-Translator

# TODO: Enter your API key
export WANDB_API_KEY=

The model and tokenizer paths default to the local copies created in the download stage and need no change:

MODEL_INPUT_PATH=/models/GaMS3-12B
TOKENIZER_PATH=/models/GaMS3-12B-Instruct

Submitting the job

sbatch sft/run_sft.sbatch \
    --lora_rank=64 \
    --warmup_steps=200 \
    --learning_rate=2e-5 \
    --min_lr=1e-6

The job requests 1 node, 4 GPUs, 32 CPUs, exclusive access, with an 8-hour limit. The four arguments follow the --key=value convention that all the training scripts in this repository use, and they are also what names the experiment directory: the run above writes to sft/experiments/r-64_ws-200_lr-2e-5_min-lr-1e-6/. That naming is what makes the merge step's checkpoint argument readable, and it means two hyperparameter settings never overwrite each other's checkpoints.

Watch the validation loss. Because the stage is short and the dataset small, overfitting is a real possibility, and it is one you can act on: checkpoints are saved twice per epoch, and the merge step takes any checkpoint, not only the last one.

When the job finishes, sft/experiments/<version>/checkpoint-<step>/ holds the LoRA adapter. Merge it into the base model (see the next chapter).

Merging LoRA Adapters

This step runs after every training stage. SFT, DPO and GRPO all save adapters only, and each stage expects a complete standalone model as its input.

The mechanics are exactly those of the PEFT service's Merging the LoRA Adapter into the Base Model chapter: load the base model in bfloat16, attach the adapter with PeftModel.from_pretrained, call merge_and_unload() to fold the low-rank update into the base weights, and save model and tokenizer together so the output directory is self-contained.

Why merging rather than stacking adapters

The pipeline could in principle keep three adapters and apply them together at inference time, or train each stage's adapter on top of the previous adapter without merging. Instead, we merge the adapter after each training stage and create a new base model for the next stage.

We do this for practical reasons. It is easier to merge adapters sequentially than to manage multiple adapters. Moreover, GRPO's inference server has to serve the current policy. TRL keeps the vLLM server in sync by merging the adapter, pushing the updated weights and unmerging. This mechanism is already delicate with one adapter.

The cost is disk: each merged 12 B model is roughly 24 GB, and the pipeline produces three of them plus the base. Budget about 100 GB of project storage, and note that you can delete a stage's merged model once the next stage's merge has succeeded.

Running the merge

merge_lora/run_merge.sbatch takes five positional arguments:

1. base_model_path - the model the training stage started from
2. tokenizer_path - tokenizer for the merged model (may equal the base model path)
3. training_dir - the repository subdirectory of the stage: sft, dpo/training, or grpo
4. adapter_checkpoint - checkpoint path relative to <training_dir>/experiments/
5. output_name - directory name for the merged model under models/

The third argument is the one that has no counterpart in the PEFT service, and it exists because this repository has three training directories rather than one. Get it wrong and the job looks for a checkpoint under the wrong stage's experiments directory.

After SFT:

sbatch merge_lora/run_merge.sbatch \
    /models/GaMS3-12B \
    /models/GaMS3-12B-Instruct \
    sft \
    r-64_ws-200_lr-2e-5_min-lr-1e-6/checkpoint-1234 \
    GaMS3-12B-SFT-Translator

Note the tokenizer argument: SFT started from the pretrained model, which has no chat template, so the tokenizer comes from the Instruct repository. This is the same base/tokenizer pairing the PEFT service uses, and merging them here is what makes every later stage simpler.

After DPO:

sbatch merge_lora/run_merge.sbatch \
    /models/GaMS3-12B-SFT-Translator \
    /models/GaMS3-12B-SFT-Translator \
    dpo/training \
    beta-0.1_r-64_ws-100_lr-5e-6_min-lr-1e-7/checkpoint-567 \
    GaMS3-12B-DPO-Translator

From here on the base and tokenizer paths are the same, because the merged SFT model carries its own tokenizer - saved alongside the weights by the previous merge.

After GRPO: the same pattern, with grpo as training_dir, the DPO model as the base, and a name such as GaMS3-12B-GRPO-Translator.

The merged model is written to models/<output_name>/ and the log to merge_lora/logs/<output_name>.txt. The job uses a single GPU with 64 GB of RAM and a 30-minute limit, which is comfortable - the merge is one pass of matrix additions, not an iterative optimization.

Before submitting, set WORK_DIR in merge_lora/run_merge.sbatch.

End-of-turn configuration

The PEFT service ends its merge chapter with a fix that is easy to miss and produces a very confusing symptom: a merged model that never stops generating. This happens because SFTTrainer's tokenization does not put the EOS token in the training sequence, and GaMS was in fact trained to end a turn with <end_of_turn> rather than EOS. The fix is to add <end_of_turn> token ID to eos_token_id in the merged model's generation_config.json. For the detailed description see the PEFT Service

Different token IDs

Different GaMS models use different token IDs for the <end_of_turn> token. GaMS3 uses ID 106, while GaMS2 uses 107. Check the tokenizer configuration for the correct ID.

Stage 2: Direct Preference Optimization

This is where the pipeline stops resembling the PEFT service with some exceptions. LoRA, ZeRO-3, Flash Attention, the schedule, the batch arithmetic are the same, and are not repeated. We explain everything else in this chapter.

What is DPO

Direct Preference Optimization trains a model on pairs of responses to the same prompt, one marked chosen and one rejected. For each pair it increases the log-probability of the chosen response relative to the rejected one, while a KL term holds the policy close to a reference model. The strength of that constraint is the beta parameter. Higher beta means less deviation from the reference.

What makes DPO attractive is its efficiency. Classical RLHF needs a separate reward model: you train a model to predict human preference, then optimize the policy against that model with a reinforcement-learning algorithm. DPO shows that for the specific case of pairwise preferences the reward model can be eliminated. The optimal policy under a preference model has a closed form, and one can optimize the policy directly on the pairs with a loss that looks much like classification (supervised approach). One model, one loss, no sampling loop. That is why this stage costs four hours and the GRPO stage costs days.

The price is that DPO is offline: it learns from a fixed dataset of pairs, constructed before training starts. It can only teach a distinction that someone has already put into the data. The chosen and rejected pairs can be produced independently of the model. However, the DPO is the most effective when the training pairs are produced by the model we want to optimize. This is called On-Policy DPO and we describe its benefits later.

The reference model costs nothing

DPO needs a reference model to measure deviation against. This is a standard approach for all preference optimization methods. The reference model is usually the frozen version of the trained model to prevent the model from diverging too far from the starting point. The naive implementation keeps a second frozen copy of the model and doubles the memory for a 12 B model (the reference model has to remain unchanged during the training, so we need two copies of weights - the ones that are updated during the training and the ones that are frozen). Our pipeline avoids that entirely:

trainer = DPOTrainer(
    model=model,
    ref_model=None,
    processing_class=tokenizer,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    peft_config=lora_config
)

Unlike full-parameter tuning LoRA keeps trained model's weight frozen and adds adapters. Consequently, same frozen weights can be used for both reference and policy models. Setting ref_model=None combined with peft_config makes DPOTrainer does exactly that. This is an elegant consequence of parameter-efficient tuning, and it is the reason preference optimization on a 12 B model is feasible on four GPUs. It applies to the GRPO stage as well.

On-policy generation: why the model has to translate first

DPO learns from the difference between chosen and rejected. If the rejected responses come from somewhere other than the model being trained - a weaker system, a corrupted reference, a synthetic corruption - then the difference the model learns is the difference between itself and that other thing, which is not what we want. What we want is for the rejected half of each pair to be a mistake this model actually makes. That is what on-policy means.

So the stage begins by having the SFT model translate a thousand Wikipedia articles and a thousand news articles, and its own systematic errors become the rejected half of each pair.

Which errors? After SFT, three recur:

1. Double newlines (paragraph separators) are collapsed to single newlines.
2. The model does not stop at the end of the document and keeps generating.
3. Dates are left in English format.

All three are mechanically repairable. Given the English source, we can restore the paragraph breaks, truncate to the source's paragraph count, and rewrite the date. The repaired version becomes chosen and the raw output stays rejected. The only difference between the two is the behaviour we want to change. This is a clean and simple training signal, because the pairs differ in exactly one respect and nothing else. This is why 1,500 pairs are enough.

Note what this stage does not teach: translation quality. Chosen and rejected contain the same words in the same order - the repair only touches structure. This is intentional and it is what the conservative beta=0.1 protects: we do not want this stage moving translation quality in either direction, only formatting.

Input documents

Two source files ship with the repository, 1,000 documents each:

data/dpo_data/
├── wikipedia_en.jsonl   # {"id": "wikipedia_0", "text": "# Title\n\n<document>"}
└── ccnews_en.jsonl      # {"id": "cc_news_0",   "text": "# Title\n*2020-06-30*\n\n<document>"}

The difference in shape is the reason there are two preparation scripts rather than one: news articles carry a date line under the title, Wikipedia articles do not.

Running the generation

Set WORK_DIR in dpo/get_translations/run_translation.sbatch, then run it once per corpus:

sbatch dpo/get_translations/run_translation.sbatch GaMS3-12B-SFT-Translator 32 wikipedia
sbatch dpo/get_translations/run_translation.sbatch GaMS3-12B-SFT-Translator 32 ccnews

The three positional arguments are the model name (a directory under models/), the batch size, and the corpus. The job uses one node with 4 GPUs and a 2-hour limit; tensor parallelism is set automatically from --tp_size=$SLURM_GPUS_ON_NODE. Logs go to dpo/get_translations/logs/<model>/<corpus>_translation.txt.

translate.py is the vLLM batch-inference pattern from Part 1: load the model once, translate in batches with model.chat(), and write results immediately after each batch so an interrupted job keeps its partial output. Generation is greedy (temperature=0) for reproducibility and to make sure we fix the mistakes that have the strongest signal in the model's output distribution (the most probable tokens). Each document produces a pair of files named by its position in the input:

data/dpo_data/<model>/<corpus>_translated/
├── 0_en.txt   # English source
├── 0_sl.txt   # Slovene translation
├── 1_en.txt
└── ...

English source is written alongside the translation so the preparation scripts are simple file-pair processors.

No resume logic

Re-running the generation job restarts from index 0 and overwrites. With 1,000 documents across 4 GPUs this comfortably fits the time limit, so it has not been worth adding; just do not assume a killed job can be continued.

Building the preference pairs

dpo/prepare_data/run_preparation.sbatch chains four steps. Only step 3 runs inside the container (COMET needs the GPU and the comet package) while the other three use nothing but the Python standard library and run directly on the node.

1. prepare_wikipedia.py reads the <idx>_en.txt / <idx>_sl.txt pairs and builds one preference record per document.
2. prepare_cc_news.py does the same for news, plus header handling.
3. compute_comet.py scores every chosen translation with wmt22-cometkiwi-da.
4. select_training_data.py drops low-scoring records, wraps both sides in chat format, shuffles and splits.

Steps 1 and 2: repair and pair

Let us describe the logic in prepare_wikipedia.py first. For each pair the script splits both texts into paragraphs, compares the paragraph structure, and then makes one of three decisions:

• Already correct: the translation has the right paragraph separators and the right paragraph count. The record is skipped: there is no difference to learn from.
• Truncated: the translation has fewer paragraphs than the source. The record is dropped. A truncated translation cannot be repaired, because the missing content does not exist. There is no way to construct a correct chosen example from it.
• Repairable: anything else. The raw translation becomes rejected, and chosen is built by restoring double newlines between paragraphs and truncating to the English paragraph count.

The repair itself is a paragraph-by-paragraph re-alignment:

def insert_double_newlines(en_paras, sl_paras):
    """Insert double newlines into text."""
    sl_iterator = iter([p for p in sl_paras if p.strip() != ""])
    sl_corrected = []
    for p in en_paras:
        if p.strip() == "":
            sl_corrected.append("")
        else:
            sl_corrected.append(next(sl_iterator))

    return sl_corrected

The English source is used as the template: wherever it has a blank line, the repaired Slovene gets a blank line, and non-empty Slovene paragraphs are consumed in order to fill the rest. Truncating to len(en_paras) then discards anything the model generated after the document ended. The title is forced to start with # if it does not already.

prepare_cc_news.py adds the date. English news documents carry a *YYYY-MM-DD* line under the title; Slovene convention is *D. M. YYYY* with no leading zeros:

def convert_date(en_date: str):
    en_date = en_date.replace("*", "").strip()

    # Convert YYYY-MM-DD to D. M. Y
    year, month, day = en_date.split('-')

    # Convert to integers to remove leading zeros and return in D. M. Y format
    return f"*{int(day)}. {int(month)}. {year}*"

Both scripts print a summary at the end: how many pairs were processed, how many had formatting errors, how many were missing double newlines, how many had extra content because the model did not stop, and how many were dropped as truncated. The two summaries differ in one line: the Wikipedia script additionally counts documents whose title was not a # heading, and the CC-News script counts documents whose date was left in English format. Read those summaries. They are a direct measurement of what your SFT model actually does wrong, and if they do not match the three errors described above, your DPO data is teaching something other than what this chapter claims.

Step 3: COMET filtering

We added COMET filtering stage as a safeguard. If we base chosen examples production only on paragraph count, there might be examples were certain paragraphs are repeated or omitted and we do not detect it. Including such pairs would be harmful even though we are not teaching the model the translation quality at this stage. Hence, every chosen candidate is scored with CometKiwi and the low scorers are removed.

CometKiwi is reference-free (it estimates the translation quality from the source and the translation alone and does not require a ground truth example), which is the property that makes it usable on data we generated ourselves. The score is added as a comet_score field.

Step 4: threshold, format and split

select_training_data.py drops records below the COMET threshold, wraps each side in chat format with the translation prompt, shuffles with a fixed seed, and creates a 90/10 train/validation split. The output records are complete preference pairs:

{
  "id": "wikipedia_42",
  "chosen":   [{"role": "user", "content": "<prompt>"}, {"role": "assistant", "content": "<repaired>"}],
  "rejected": [{"role": "user", "content": "<prompt>"}, {"role": "assistant", "content": "<raw>"}],
  "comet_score": 0.83
}

The <prompt> is identical for both chosen and rejected. It is built the same way as in the SFT data: PROMPT_TEMPLATE, two newlines and the English document.

Running it

Set WORK_DIR in dpo/prepare_data/run_preparation.sbatch, confirm the COMET checkpoint is at translation_reward_models/wmt22-cometkiwi-da/checkpoints/model.ckpt, then:

sbatch dpo/prepare_data/run_preparation.sbatch --model=GaMS3-12B-SFT-Translator

The --model value must be the same model name passed to the generation job.

Argument	Default	Description
--model	(required)	Model whose translations to process
--wikipedia_comet_threshold	0.65	Minimum COMET score to keep a Wikipedia pair
--ccnews_comet_threshold	0.7	Minimum COMET score to keep a CC-News pair
--seed	42	Random seed for shuffling

The two thresholds differ because the average COMET score on the news corpus is slightly higher. Raising the default values trades data volume for data quality. select_training_data.py prints how many examples survived, and this number is worth checking on every run. The filter chain is aggressive: already-correct documents skipped, truncated documents dropped, low-COMET documents filtered. It is possible to start with 2,000 documents and end with a few hundred pairs.

The job requests 1 GPU with a 1-hour limit, and logs to dpo/prepare_data/logs/<model>_preparation.txt. Output:

data/dpo_data/<model>/dpo_training/<model>/
├── wikipedia.jsonl         # step 1
├── ccnews.jsonl            # step 2
├── wikipedia_comet.jsonl   # step 3
├── ccnews_comet.jsonl      # step 3
├── training.jsonl          # step 4  <- DPO input
└── validation.jsonl        # step 4  <- DPO input

Training

Open dpo/training/run_dpo.sbatch and fill in:

WORK_DIR=/path/to/GaMS-Translator

# TODO: Set the name of the SFT-trained model (a dir inside $MODELS_DIR) that DPO starts from.
INPUT_MODEL_NAME=GaMS3-12B-SFT-Translator

# TODO: Point this at the dir with training.jsonl + validation.jsonl produced by dpo/prepare_data
DATA_DIR=$WORK_DIR/data/dpo_training

# TODO: Enter your API key
export WANDB_API_KEY=

Make sure you do not overlook theDATA_DIR TODO. The preparation job writes the training data into a nested per-model directory. The default value here points only to the root of nested directories. Point it at the directory that actually contains training.jsonl and validation.jsonl files.

sbatch dpo/training/run_dpo.sbatch \
    --beta=0.1 \
    --lora_rank=64 \
    --warmup_steps=100 \
    --learning_rate=5e-6 \
    --min_lr=1e-7

The job requests 2 nodes × 4 GPUs, 32 CPUs per task, exclusive access, 4-hour limit.

The configuration choices that differ from SFT:

• Learning rate 5e-6, minimum 1e-7: smaller learning rates than during SFT. While SFT is teaching a task, DPO is nudging a model that already performs it. Hence, a learning rate appropriate for SFT could potentially harm the model during the DPO training.
• beta=0.1: conservative, as discussed above.
• No Liger kernel. See the note at the end of the pipeline overview chapter.

When training finishes, merge the adapter to produce GaMS3-12B-DPO-Translator.

At this point you have a model that translates as well as the SFT model but formats correctly: right paragraph structure, stops at the end, Slovene dates. That is a real improvement in usability and no improvement at all in translation quality. We improve the latter in the next (and last) stage.

Stage 3: Group Relative Policy Optimization

This is the final stage in our pipeline and consequently the stage the models are named after. In this stage we improve the translation quality through online preference optimization. Consequently, this stage requires a lot of compute time (multiple days/weeks on twelve GPUs).

What GRPO is

Group Relative Policy Optimization is an online reinforcement-learning method. For each prompt the model generates a group of candidate completions (rollouts). Each candidate is scored by one or more reward functions. The reward function of each candidate is compared to the group's mean. Based on these comparisons, the advantages are computed. The policy is updated to favor the candidates with higher advantages (the candidates that perform better than the rest of the group).

Group-based optimization is an important part of the GRPO method. Classical policy-gradient methods (such as PPO better known as reinforcement learning with human feedback) need a value model (a second network trained to predict the expected return from a state) in order to turn a reward into an advantage. A reward of 0.8 is meaningless without knowing whether 0.8 is good for this input. GRPO replaces the value network with the group. It samples multiple translations of the same document, and the mean of their scores is a perfectly good estimate of "what this model typically achieves on this input". A candidate above the mean gets a positive advantage, one below it a negative one. No critic, no second network, no extra memory.

The advantage of GRPO over DPO is its online approach. DPO learns from a fixed dataset of pairs that someone constructed in advance. It can only teach differences that are already in the data. GRPO generates fresh samples from the current policy at every step and scores them with a function, which means it can teach any distinction the function can measure - including "this translation is better than that one" in a sense no repair script could produce. That is why it improves quality where DPO could not, and why it needs a live inference server.

Why is it expensive

Every optimizer step in GRPO waits on generation. With a global batch of 64 prompts and 4 completions per prompt, a single step requires 256 full document translations to be generated, and then all 256 to be scored by a neural quality metric. Since generation is the bottleneck in terms of speed, TRL's GRPO Trainer supports vLLM integration. This significantly speeds up the generation, but generating 256 translations at once can still take a few minutes.

The vLLM integration separates the generation model and the trainer. GRPO Trainer sends weight updates to the vLLM model after each training step. GRPO Trainer supports two different vLLM integrations (controlled by vllm_mode argument):

• collocate: vLLM model shares the GPUs with the Trainer
• server: vLLM model is set up as a separate server

In this service we choose the second option for the memory reasons. As trainer requires a large proportion of vRAM available at LEONARDO A100 GPUs, we need to set up vLLM server separately to avoid CUDA OOM issues. We set up the server on a separate node.

The job requests three nodes and assigns roles by SLURM rank:

Rank	Role
0, 1	Trainers - 8 GPUs total, DeepSpeed ZeRO-3
2	Inference server - trl vllm-serve, tensor parallel across its 4 GPUs

The trainers reach the server over HTTP at http://<node2>:8000, which is passed to the training script as --vllm_url. TRL keeps the server's weights in sync as training progresses, and for a LoRA model it does so by merging the adapter into the base weights, pushing the updated weights to the server, and then unmerging.

The role assignment lives in grpo/run_grpo.sbatch:

NODELIST=($(scontrol show hostnames $SLURM_JOB_NODELIST))

MASTER_ADDR=${NODELIST[0]}
TRAINER_NODE_1=${NODELIST[0]}
TRAINER_NODE_2=${NODELIST[1]}
INFERENCE_NODE=${NODELIST[2]}

if [ "$MY_RANK" -eq 2 ]; then
    trl vllm-serve \
        --model /model_input_path \
        --tensor-parallel-size $SLURM_GPUS_ON_NODE \
        --port $VLLM_PORT
elif [ "$MY_RANK" -eq 0 ] || [ "$MY_RANK" -eq 1 ]; then
    python3 -m torch.distributed.run \
      --nproc_per_node $SLURM_GPUS_ON_NODE \
      --nnodes 2 \
      --node_rank $MY_RANK \
      ...
    scancel $SLURM_JOB_ID
fi

Two details make this work. The vLLM server runs in the foreground, so the SLURM task stays alive for the duration of the job rather than exiting immediately after launching a background process. And when the trainers finish, they call scancel on their own job - otherwise the inference node would sit holding four GPUs until the four-day wall clock expired, because nothing else would ever tell it to stop.

If you adapt this topology, the ratio of trainer nodes to inference nodes is the knob to turn. Two trainers and one server is balanced for a 12 B model and document-length generation. A larger model might need more trainer nodes. Adding a second or third generation node might speed-up the generation. However with smaller batch sizes the gains are diminishing (the speed-up comes from the number of examples that can be generated in parallel, which is limited by the batch size).

The reward functions

grpo/reward_functions.py defines four rewards, combined with the weights in reward_weights:

Reward	Weight	What it measures
comet_score	1.0	CometKiwi quality of the translation given the English source (reference-free)
language_score	0.2	fastText probability that the output is Slovene
length_score	0.2	Word-count ratio ([0, 1] interval) against the source. Peaks when lengths match
bleu_score	0.1	sacreBLEU against the reference translation (produced by different models)

COMET dominates because it measures the quality of the translation without using the reference. This is our core optimization objective. The other three rewards exist as a safeguard against specific ways a policy can score well on COMET while getting worse (reward hacking). Language score is there to prevent the model from returning the output in the source language (English). Length score is there to make sure the model does not truncate/expand the translation. The BLEU score is there as a regularization metric (in addition to KL beta) that prevents the model from deviating too far away from provided "ground truth" machine translations.

COMET Score

The main signal. CometKiwi is reference-free, taking the source and the translation and returning a quality estimate. The model is loaded once per process and cached:

@lru_cache(maxsize=1)
def get_comet_model() -> comet.models.CometModel:
    model_path = "/reward_models/wmt22-cometkiwi-da/checkpoints/model.ckpt"
    model = load_from_checkpoint(model_path, reload_hparams=True, local_files_only=True)
    return model

Without the lru_cache decorator a COMET model would be loaded from disk on every reward call, several times per step.

Reward computation requires the source text, but the reward function is given the full prompt instead. Therefore we need to extract the source from the prompt by removing the instruction prefix:

def extract_src(prompt: str) -> str:
    source = prompt.removeprefix(PROMPT_TEMPLATE)
    return source.strip()

This one-line function is the reason the instruction has to be identical everywhere in the pipeline. More on that below.

Language Score

FastText's lid.176.bin language identifier outputs probability that the output is in certain language for each language in its output layer (multi-class classification). We take the probability that the completion is Slovene as our reward. It protects the model from returning the English original or drifting into Croatian, Serbian or Bosnian, which are close enough that a quality metric may not object strongly. Completions shorter than 3 characters score 0, since language identification on a fragment is unreliable.

Length Score

A word-count ratio between the completion and the source, shaped to peak at parity:

    for sol, source in zip(solutions_len, source_len):
        ratio = float(sol / source)
        if 1.0 < ratio < 2.0:
            score = 2.0 - ratio
        elif ratio >= 2.0:
            score = 0.0
        else:
            score = ratio
        scores.append(score)

Below parity the score is the ratio, so a translation half the length of the source scores 0.5. Above parity it falls off linearly and hits zero at twice the length. This penalizes the two remaining structural failures from opposite directions: truncating the document, and rambling past its end. It is a crude measure (Slovene and English do not have identical word counts for the same content) but it does not need to be precise, only to make the extremes unattractive.

BLEU Score

BLEU score against a reference translation supplied in the dataset's chosen column, divided by 100 to bring it onto the same 0–1 scale as the others. Its weight of 0.1 reflects how much we trust it: BLEU is sensitive to exact word choice and segmentation, and a fluent Slovene translation that restructures a sentence is punished by it. It is included as a weak anchor, not as a target.

Input data

We use publicly available cjvt/GaMS-Translator-GRPO-Training dataset, downloaded in the environment-setup stage. gams_grpo.py reads the parquet shards directly:

dataset = load_dataset("parquet", data_files={
    "training":   "/data/training-*.parquet",
    "validation": "/data/validation-*.parquet",
})

We use globs rather than exact filenames, so the dataset can be re-sharded upstream without touching the code. Note the split names are training and validation, not the usual train.

Split	Examples (before filtering)
training	103,351
validation	1,044

Column	Type	Used for
prompt	list of {role, content}	The instruction + English document to translate
chosen	list of {role, content}	Reference translation - the target for bleu_score
comet_score	float	COMET score of the reference, from dataset construction
translator	string	Which system produced the reference (gams2, gams3, gemini, translate-gemma)

comet_score and translator are not used during the training. However, they are passed through to the reward functions (due to remove_unused_columns=False) as keyword arguments and absorbed by their **kwargs, so they are available if you want to weight or filter on them. For example, you could discount the BLEU reward when the reference itself is scored poorly.

Instruction filtering, and why it exists

The dataset is not fully homogeneous. Most rows use the long "profesionalni prevajalec" instruction that reward_functions.PROMPT_TEMPLATE defines, but a small number (some examples with translator == "gemini") use a shorter one: Prevedi naslednje besedilo v slovenščino.

This is a larger problem than just a cosmetic inconsistency. extract_src recovers the source by stripping PROMPT_TEMPLATE, and str.removeprefix is a no-op when the prefix does not match. It does not raise any warning or error, it just returns the string unchanged. Consequently, such row would keep a Slovene instruction line inside its "English source text", which skews the COMET, length and BLEU rewards.

gams_grpo.py drops these rows immediately after loading:

raw_sizes = {split: len(split_dataset) for split, split_dataset in dataset.items()}
dataset = dataset.filter(has_expected_instruction)

The has_expected_instruction function imports PROMPT_TEMPLATE from reward_functions, so the filter and the extraction can never disagree. On the validation split this removes 26 of 1,044 rows, leaving 1,018. Both counts are reported on rank 0 at startup:

Train data size: <kept> (dropped <n> with unexpected instruction)
Val data size: 1018 (dropped 26 with unexpected instruction)

Watch those numbers on the first run. A large drop means the dataset's instruction format has diverged from PROMPT_TEMPLATE, and finding that out in the first minute is way better than finding it out after four days of GPU time. If you change the prompt, this filter will report it as a total wipe-out of the dataset.

Configuration

sbatch grpo/run_grpo.sbatch \
    --beta=0.04 \
    --lora_rank=64 \
    --warmup_steps=100 \
    --learning_rate=5e-6 \
    --min_lr=5e-7

Before submitting, fill in grpo/run_grpo.sbatch:

WORK_DIR=/path/to/GaMS-Translator

# TODO: Set the name of the DPO-trained model (a dir inside $MODELS_DIR) that GRPO starts from.
INPUT_MODEL_NAME=GaMS3-12B-DPO-Translator

# TODO: Enter your API key
export WANDB_API_KEY=

DATA_DIR needs no change - it already points at the dataset downloaded earlier:

DATA_DIR=$WORK_DIR/data/grpo_training/data

The job requests 3 nodes × 4 GPUs, --qos=boost_qos_lprod, and a 4-day limit (--time=4-00). That QoS exists on LEONARDO specifically to allow long-running jobs (the default maximal time limit for a job on LEONARDO is 1 day). The 4-day limit is set simply because this is the maximal time limit allowed on LEONARDO. In reality this training stage might take few weeks on this amount of data and with this number of rollouts. We trained the published models on FRIDA supercomputer on 2 DGX B300 nodes. On LEONARDO, you will either have to resume the training multiple times or significantly reduce the training data size. This is why we published our models and suggest using this stage only if nothing else works.

The settings that matter:

• num_generations=4: the group size (number of rollouts). This is the parameter that makes the advantage estimate possible Four is a small group (pragmatic floor) and the mean of four samples is a noisy baseline. Larger groups give a better estimate and result in proportionally larger generation times.
• **batch_size=64: number of prompts (examples) in a single training steps. With num_generations=4 this results in 256 model generations at each training steps. GRPO Trainer takes the number of generations in account while computing the batch size. Hence, we define the gradient accumulation steps as product of batch size and number of generations divided by micro batch size and world size:

gradient_accumulation_steps = batch_size * num_generations // (world_size * micro_batch_size)

• temperature=0.1: low, for a task where the goal is one correct output rather than diversity. However, we need to keep in mind that GRPO needs the group members to differ in order to have anything to compare. This can be problematic with a low temperature. For a task with a broader space of good answers (reasoning, code, open-ended generation) a higher temperature is usually necessary to make the group informative.
• generation_kwargs={"stop_token_ids": [1, 106]}: the same EOS/<end_of_turn> pairing discussed in the merge chapter. Without it, generation runs to max_completion_length on every sample.
• beta=0.04: lower than DPO's 0.1, so the policy is allowed to drift further from its reference. Improving quality requires moving. However, we added the safeguards (different reward functions) to keep the movement sane.
• max_prompt_length=4096, max_completion_length=4096: tighter than the 8,192 of the earlier stages, because here both are measured separately.
• mask_truncated_completions=False: a truncated completion still receives its reward, and the length reward will score it poorly. The alternative, masking them out, would remove the training signal that discourages producing them. Additionally, the training marks stopping on "stop_token_ids": [1, 106] passed through generation_kwargs as truncations, meaning we need to keep this setting to false.
• 3 epochs: evaluation four times per epoch, checkpointing eight times per epoch. We need to checkpoint more often due to longer training time.
• No Liger kernel. See the note at the end of the pipeline overview chapter.

Consider also setting log_completions=True in GRPOConfig (if your TRL version does not already default to it), so that sample generations appear in the WandB dashboard. Reading actual completions is the only practical way to notice reward hacking while there is still time to stop the job.

eval_on_start=True

This triggers a full generation pass over the validation set before the first training step. It gives you a baseline to compare against, which is genuinely useful, but it is not "free" for GRPO. It means generating and scoring 1,018 translations before any learning happens. Remove the flag if you would rather not "pay" for it.

Adapting the Pipeline to Another Task

Machine translation is the working example in this service, but this pipeline can be adapted for preference-optimizing an LLM on different tasks. In this closing chapter we discuss what has to change if we change the task.

The question to answer first

Before any code changes, answer this: can you score a candidate output automatically? The answer to this question decides which parts of the pipeline you can use.

If the answer is yes (you have a metric, a validator, a test suite, a classifier, a rule set, anything that takes a candidate and returns a number) then the GRPO stage is available to you, and it is the most powerful stage of this service. Translation happens to have an automatic scorer in CometKiwi, but you do not necessarily need "a neural quality metric". A reward function can be a compiler exit code, a schema validator, a regular expression, an exact-match check against a known answer, or a small classifier you trained yourself. It has to be cheap enough to run hundreds of times per optimizer step, and it has to be reliable.

If the answer is no, you can still use SFT and DPO. However, DPO then needs preference pairs from somewhere, and constructing them is the main challenge. The two honest routes are human annotation, which is expensive and slow, and programmatic repair of the kind this pipeline uses, which requires that the errors you want to fix are mechanically correctable. If neither is available, preference optimization is not the right tool and the PEFT service's supervised fine-tuning is likely to serve you better.

Changing the language pair or the domain

This section applies only if you are using the pipeline for translation task, but for other pair of languages. This is the smallest change and touches mostly data.

• The prompt. Rewrite PROMPT_TEMPLATE for your task and change it in all four places it appears: the SFT data, dpo/get_translations/translate.py, dpo/prepare_data/select_training_data.py, and grpo/reward_functions.py. The instruction-filter in gams_grpo.py will tell you if the GRPO data disagrees with the constant (see the GRPO chapter).
• The base model. For a language other than Slovene, GaMS is the wrong starting point. Use a strong open model for the target language. Everything else in the pipeline is model-agnostic apart from the <end_of_turn> token id 106, which is specific for Gemma 3 and appears in the merge-time generation config and in GRPO's stop_token_ids. Find the equivalent token for your model.
• The language reward. language_score uses fastText's lid.176.bin, which covers 176 languages. Changing the target is changing the label from __label__sl.
• The COMET reward. CometKiwi is multilingual and covers many language pairs, but check that yours is among them before relying on it.
• SFT data. You need to obtain a thousand high-quality examples for your pair of languages. This part of the pipeline has no shortcut.

For a domain shift rather than a language shift (legal, medical, technical translation) the base model and rewards can stay, but generic quality may no longer be the right objective. A domain-specific reward, for instance a terminology-adherence check against a glossary, added with a small weight alongside COMET, is a natural extension and exactly the shape the reward-function interface expects.

Writing a reward function

If you adapt the pipeline for your new taks, you will have to write your own reward function(s). The interface is a plain Python function returning one score per completion:

def my_reward(completions: List, prompts: List, chosen: List, **kwargs) -> List:
    return [score_one(c[0]["content"]) for c in completions]

Register it in the reward_funcs list passed to GRPOTrainer and give it a weight in reward_weights. The two lists are positional and must line up (the first weight in the weights list is the weight of the first function from the reward functions list and so on). If you do not have the ground truth examples, simply remove the chosen argument from the function's definition. The **kwargs absorbs any extra dataset columns, which are available because remove_unused_columns=False. You can use these columns (or prompts) if your reward function requires more inputs that just pure completion.

Return values on a common scale. All four rewards here return a number between 0 and 1, which is why the weights are interpretable as relative importance. bleu_score divides sacreBLEU's 0–100 output by 100 for exactly this reason. Mixing scales means your weights are doing arithmetic you did not intend.

Cache anything expensive. @lru_cache(maxsize=1) on the model loaders is what stops a reward model being read from disk on every call. If your reward needs a model, cache it the same way. If you are using another LLM as a reward model, try to set it up as a separate vLLM server on another node and perform API calls during the reward computation.

Assume the policy will attack the reward. Ask what the cheapest possible output that scores well would look like, and whether it is something you would accept. If it is not, you need a guard rail. The three low-weight rewards in this pipeline exist because each one blocks a specific cheap win: produce almost nothing, produce far too much, produce fluent text in the wrong language.

Fail loudly on malformed input. The instruction-filter story in the GRPO chapter is the cautionary tale: str.removeprefix silently returning an unmodified string produced a reward computed on the wrong text, with no error. Anywhere your reward function makes an assumption about the shape of the prompt or the completion, assert it. A job that dies in the first minute is far cheaper than one that trains for four days on a corrupted signal.

A realistic order of work

If you are adapting this pipeline, try to follow this order to prevent spending too much compute for no good reason:

1. Write the reward function first, and test it offline on a few hundred outputs from an existing model. Look at the score distribution. Look at the highest-scoring and lowest-scoring outputs by hand and ask whether you agree with the ranking. A reward function you have not inspected is not ready to spend GPU-days on.
2. Collect the SFT data (the thousand examples) and run the SFT stage. Merge, and check the model's outputs.
3. Generate on-policy outputs from the SFT model and catalogue the failures. This step is diagnostic, and it is what tells you whether a DPO stage is worth running at all. If the failures are not mechanically repairable, skip DPO and go straight to GRPO.
4. Run GRPO on a small model and a small slice of data, end to end, until it completes. Confirm the reward components move sensibly in WandB.
5. Only then run it at full scale.