Per-Prompt LLM Selection — Automatically Choosing the Best Open Access LLM
Opis
This service automatically chooses the best open-access available AI model to answer each user's query. Instead of relying on a single model for all tasks, it analyzes each prompt and routes it to the model most likely to provide the best response. By combining the strengths of multiple open-access AI models, the system delivers more accurate, efficient, and reliable results across different types of questions. The approach is grounded in supervised and unsupervised meta-learning: prompt embeddings generated by a diverse set of embedding models are jointly clustered, and the resulting cluster structure is used to recommend the best-performing LLM for each incoming prompt. Benchmark performance data is sourced from the Holistic Evaluation of Language Models (HELM) study, which provides standardised evaluations across diverse tasks including question answering, summarisation, reasoning, translation, and information extraction.
Pričakovani rezultati
Given an input prompt, the meta-model outputs a recommendation identifying the publicly available LLM most likely to produce the best response for that prompt. The system is evaluated against the VBS baseline across both within-dataset and cross-dataset scenarios.
Kartica storitve
| Kategorija | AI development and deployment |
| Domena | AutoML, Meta-Learning, Algorithm Selection, Natural Language Processing |
| Vertikala | Cross-domain |
| Ciljni uporabniki | ML Engineers and Application Developers Seeking to Automatically Route Prompts to the Most Suitable Publicly Available LLM |
| Tip storitve | Meta-model |
| Rezultat | Per-prompt LLM recommendation. |
| Zahteve | Pre-computed embedding for the new prompts; GPU. |
| Omejitve | Meta-model recommendations are limited to LLMs for which complete performance data is available across all analysed datasets; recommendation quality depends on the diversity and representativeness of the selected embedding models and benchmark datasets. |
| Na voljo od | Q3/2027 |
| Licenca | MIT and CC BY-NC-SA 4.0 |
| Slika | assets/service-visuals/per-prompt-llm-selection-automatically-choosing-the-best-open-access-llm.png |
| Oznaka slike |