LLM Providers

The LLM provider configuration defines the credentials used for evaluation judging and persona generation. The provider is stored inline in evalstudio.config.json under the llmSettings key — there is no separate entity or storage file.

Import

import {
  getLLMProviderFromProjectConfig,
  getDefaultModels,
  type LLMProvider,
  type LLMSettings,
  type LLMModelSettings,
  type ModelGroup,
  type ProviderType,
} from "@evalstudio/core";

Types

ProviderType

type ProviderType = "openai" | "anthropic";

LLMSettings

Unified LLM configuration stored in evalstudio.config.json.

interface LLMSettings {
  provider: ProviderType;          // Provider type (openai or anthropic)
  apiKey: string;                  // API key for the provider
  models?: LLMModelSettings;      // Model selection per use-case
}

LLMModelSettings

interface LLMModelSettings {
  evaluation?: string;  // Model for evaluation/judging (optional, uses provider default)
  persona?: string;     // Model for persona generation (optional, falls back to evaluation)
}

ModelGroup

Models are organized into groups (e.g., Standard, Premium) per provider.

interface ModelGroup {
  label: string;     // Group name (e.g., "Standard", "Premium")
  models: string[];  // Model IDs in this group
}

LLMProvider

Runtime representation used internally by the evaluator and persona generator.

interface LLMProvider {
  provider: ProviderType;
  apiKey: string;
  config?: LLMProviderConfig;
}

Functions

getLLMProviderFromProjectConfig()

Reads the effective project config and returns an LLMProvider object.

async function getLLMProviderFromProjectConfig(
  storage: StorageProvider,
  workspaceDir: string,
  projectId: string,
): Promise<LLMProvider>;

Throws: Error if no LLM provider is configured in evalstudio.config.json.

const provider = await getLLMProviderFromProjectConfig(storage, workspaceDir, projectId);
console.log(provider.provider);  // "openai"

getDefaultModels()

Returns grouped model lists for each provider type, organized by tier (Standard, Premium).

function getDefaultModels(): Record<ProviderType, ModelGroup[]>;

const models = getDefaultModels();

// Each provider returns an array of ModelGroup
for (const group of models.openai) {
  console.log(group.label);   // "Standard" or "Premium"
  console.log(group.models);  // ["gpt-4o", "gpt-4o-mini", ...]
}

Configuration

The LLM provider is configured in evalstudio.config.json:

{
  "name": "my-project",
  "llmSettings": {
    "provider": "openai",
    "apiKey": "sk-your-api-key",
    "models": {
      "evaluation": "gpt-4o",
      "persona": "gpt-4o-mini"
    }
  }
}

See Projects for the full config reference.