Model Evaluation for the Local Context Using Linguistic and Cultural Competence Measures
Overview
This service provides novel methodologies and high-quality datasets for evaluating and benchmarking AI models in the local Slovene context. It addresses a critical gap in existing evaluation frameworks, which are often biased toward high-resource languages and lack sensitivity to linguistic and cultural nuances specific to Slovenia. The service delivers the first two comprehensive and original benchmarks focused on Slovene linguistic and cultural competence, based on manually curated examples and previously unpublished language examination materials, ensuring both authenticity and high quality. The benchmarks contain more than 4,000 instances across various task types, including multiple-choice question answering, long-form answering, visual reasoning, and others. Strict data protection measures ensure benchmark datasets remain confidential and inaccessible to model training pipelines, preserving evaluation validity over time. The service employs targeted evaluation procedures for each task type, combining standard quantitative metrics (such as accuracy for closed-form tasks) with modern methodologies suited to generative outputs (such as LLM-as-a-judge). Benchmark construction relies on expert linguistic curation, structured annotation workflows, and quality assurance protocols. The evaluation framework supports both automated scoring pipelines and human-in-the-loop assessment for open-ended tasks.
Outcome
Users receive standardised evaluation reports assessing their AI models against Slovene linguistic competence (grammar, vocabulary, reading comprehension, stylistic register) and cultural competence (knowledge of Slovenian history, geography, institutions, customs, and societal context). Reports include per-task-type scores, comparative rankings against baseline models, and diagnostic breakdowns identifying specific areas of strength and weakness. The service enables informed model selection, targeted improvement, and compliance assessment for AI deployments in the Slovenian market.
Service Card
| Category | AI development and deployment |
| Domain | NLP, Model Evaluation, Slovenian Language |
| Vertical | Digital Society |
| Target users | AI Model Developers, NLP Researchers, LLM Providers, Public Sector Organisations Evaluating AI Systems for Slovene Compliance, Standardisation Bodies, Educational Institutions |
| Service type | Data, Framework |
| Outputs | Standardised evaluation reports with per-task scores and diagnostic breakdowns; comparative rankings; linguistic and cultural competence profiles; recommendations for model improvement. |
| Requirements | NVIDIA GPU node for model inference during evaluation. Model must support Slovene language input; multimodal models required for visual reasoning tasks. Models are submitted for evaluation through a controlled access protocol---benchmark data is never shared with model providers. |
| Limitations | |
| Availability | Q2/2027 |
| Licensing | Benchmark framework: open (MIT); benchmark datasets: restricted access (confidential, not available for model training) |
| Visual content | |
| Visual caption |