> ## Documentation Index
> Fetch the complete documentation index at: https://docs.fallom.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluations

> Run LLM evaluations locally or directly from your dashboard

Fallom Evals lets you evaluate your LLM outputs using G-Eval with an LLM-as-judge approach. You can run evaluations locally using our SDK, or directly from the Fallom dashboard on your production traces—no code required.

## Features

* **7 Built-in Metrics**: Answer relevancy, hallucination, toxicity, faithfulness, completeness, coherence, bias
* **G-Eval Methodology**: Chain-of-thought prompting for accurate scoring
* **Dashboard Evals**: Run evals on production traces directly from the UI—no SDK needed
* **Model Comparison**: Test multiple models on the same dataset
* **Custom Pipeline Support**: Evaluate outputs from your own RAG or multi-agent systems
* **Auto-Upload**: Results automatically sync to your dashboard
* **Fallom Datasets**: Use datasets stored in Fallom or create them locally

## Dashboard Evals (No Code Required)

If you're already logging traces to Fallom, you can run evaluations directly from the dashboard without writing any code. Navigate to **Evals Store → Evals** to create evaluation configs that automatically sample and evaluate your production traces.

### Creating an Eval Config

1. Go to **Evals Store → Evals** in your dashboard
2. Click **New Config**
3. Configure your evaluation:
   * **Name**: A descriptive name for your eval config
   * **Sample Rate**: Percentage of traces to evaluate (0.01% to 100%)
   * **Judge Model**: The LLM to use as the evaluator (e.g., `openai/gpt-4o-mini`)
   * **Filter by Tags**: Only evaluate traces with specific tags (optional)
   * **Filter by Models**: Only evaluate traces from specific models (optional)
   * **Metrics**: Select which metrics to run (answer relevancy, hallucination, etc.)

### Running Evaluations

Once you've created a config, click **Run Now** to start an evaluation. Fallom will:

1. Sample recent traces from the last 15 minutes matching your filters
2. Queue them for evaluation using your selected judge model
3. Score each trace against your chosen metrics
4. Display results with per-metric scores and aggregated statistics

### Viewing Results

Each evaluation run shows:

* **Sample Count**: How many traces were evaluated
* **Scores**: Average, min, and max scores for each metric
* **Individual Results**: Click on a run to see detailed scores for each trace
* **Regression Detection**: Automatic alerts when quality drops compared to previous runs

<Note>
  Dashboard evals require traces to be logged to Fallom first. Make sure you have [tracing](/tracing) set up before running dashboard evals.
</Note>

## Quick Start

<Tip>
  **No code required?** If you're already logging traces to Fallom, you can skip the SDK setup and [run evals directly from the dashboard](#dashboard-evals-no-code-required).
</Tip>

### Using the SDK

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    from fallom import evals

    # Initialize (enables auto-upload to dashboard)

    evals.init(api_key="your-fallom-api-key")

    # Create a dataset

    dataset = [
    evals.DatasetItem(
    input="What is the capital of France?",
    output="The capital of France is Paris.",
    system_message="You are a helpful assistant."
    ),
    ]

    # Run evaluation - results auto-upload!

    results = evals.evaluate(
    dataset=dataset,
    metrics=["answer_relevancy", "faithfulness", "completeness"]
    )

    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import fallom from "@fallom/trace";

    // Initialize (enables auto-upload to dashboard)
    fallom.evals.init({ apiKey: "your-fallom-api-key" });

    // Create a dataset
    const dataset = [
      {
        input: "What is the capital of France?",
        output: "The capital of France is Paris.",
        systemMessage: "You are a helpful assistant."
      }
    ];

    // Run evaluation - results auto-upload!
    const results = await fallom.evals.evaluate({
      dataset,
      metrics: ["answer_relevancy", "faithfulness", "completeness"]
    });
    ```
  </Tab>
</Tabs>

## Environment Variables

```bash theme={null}
# Required
FALLOM_API_KEY=your-fallom-api-key     # For uploading results & fetching datasets
OPENROUTER_API_KEY=your-openrouter-key # For judge model & model comparison
```

## Available Metrics

| Metric             | Description                                                 |
| ------------------ | ----------------------------------------------------------- |
| `answer_relevancy` | Does the response directly address the user's question?     |
| `hallucination`    | Does the response contain fabricated information?           |
| `toxicity`         | Does the response contain harmful or offensive content?     |
| `faithfulness`     | Is the response factually accurate and consistent?          |
| `completeness`     | Does the response fully address all aspects of the request? |
| `coherence`        | Is the response logically structured and easy to follow?    |
| `bias`             | Does the response contain unfair or prejudiced content?     |

All metrics return a score from 0.0 to 1.0, where higher is better (except hallucination, toxicity, and bias, where higher means more problematic content detected).

## Custom Metrics

Create your own evaluation metrics with custom criteria and evaluation steps:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    from fallom import evals

    evals.init()

    # Define a custom metric

    brand_metric = evals.CustomMetric(
    name="brand_alignment",
    criteria="Brand Alignment - Does the response follow brand voice guidelines?",
    steps=[
    "Check if the tone is professional yet friendly",
    "Verify no competitor brands are mentioned",
    "Ensure the response uses approved terminology",
    "Check for appropriate emoji usage (none in formal contexts)"
    ]
    )

    # Use with built-in metrics

    results = evals.evaluate(
    dataset=dataset,
    metrics=["answer_relevancy", brand_metric]
    )

    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import fallom from "@fallom/trace";

    fallom.evals.init({});

    // Define a custom metric
    const brandMetric = {
      name: "brand_alignment",
      criteria: "Brand Alignment - Does the response follow brand voice guidelines?",
      steps: [
        "Check if the tone is professional yet friendly",
        "Verify no competitor brands are mentioned",
        "Ensure the response uses approved terminology",
        "Check for appropriate emoji usage (none in formal contexts)"
      ]
    };

    // Use with built-in metrics
    const results = await fallom.evals.evaluate({
      dataset,
      metrics: ["answer_relevancy", brandMetric]
    });
    ```
  </Tab>
</Tabs>

Custom metrics use the same G-Eval methodology as built-in metrics - the LLM judge follows your steps and provides reasoning and a score.

## Using Datasets from Fallom

Instead of creating datasets locally, you can use datasets stored in Fallom. Just pass the dataset key:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    from fallom import evals

    evals.init()

    # Just pass the dataset key - it auto-fetches from Fallom!

    results = evals.evaluate(
    dataset="my-dataset-key", # Dataset key from Fallom
    metrics=["answer_relevancy", "faithfulness"]
    )

    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import fallom from "@fallom/trace";

    fallom.evals.init({});

    // Just pass the dataset key - it auto-fetches from Fallom!
    const results = await fallom.evals.evaluate({
      dataset: "my-dataset-key",  // Dataset key from Fallom
      metrics: ["answer_relevancy", "faithfulness"]
    });
    ```
  </Tab>
</Tabs>

## Evaluating Custom Pipelines

If you have a complex LLM pipeline (RAG, multi-agent, custom routing), use `EvaluationDataset` to run your own pipeline and evaluate the outputs:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    from fallom import evals

    evals.init(api_key="your-fallom-api-key")

    # Pull a dataset from Fallom

    dataset = evals.EvaluationDataset()
    dataset.pull("customer-support-qa")

    # Run each input through YOUR pipeline

    for golden in dataset.goldens: # Your custom pipeline (RAG, agent routing, etc.)
    actual_output = my_rag_pipeline(golden.input)

        dataset.add_test_case(evals.LLMTestCase(
            input=golden.input,
            actual_output=actual_output,
            context=retrieved_docs  # Optional: for faithfulness eval
        ))

    # Evaluate your outputs

    results = evals.evaluate(
    test_cases=dataset.test_cases,
    metrics=["answer_relevancy", "faithfulness", "completeness"]
    )

    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import fallom from "@fallom/trace";

    fallom.evals.init({ apiKey: "your-fallom-api-key" });

    // Pull a dataset from Fallom
    const dataset = new fallom.evals.EvaluationDataset();
    await dataset.pull("customer-support-qa");

    // Run each input through YOUR pipeline
    for (const golden of dataset.goldens) {
      // Your custom pipeline (RAG, agent routing, etc.)
      const actualOutput = await myRAGPipeline(golden.input);

      dataset.addTestCase({
        input: golden.input,
        actualOutput,
        context: retrievedDocs  // Optional: for faithfulness eval
      });
    }

    // Evaluate your outputs
    const results = await fallom.evals.evaluate({
      testCases: dataset.testCases,
      metrics: ["answer_relevancy", "faithfulness", "completeness"]
    });
    ```
  </Tab>
</Tabs>

### Auto-Generate Test Cases

For simpler pipelines, use `generate_test_cases()` to automatically run all inputs through your pipeline:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    from fallom import evals

    evals.init()

    def my_pipeline(messages):
    """Your pipeline function"""
    user_query = messages[-1]["content"]
    response = my_rag_app(user_query)
    return {"content": response.text}

    dataset = evals.EvaluationDataset()
    dataset.pull("my-dataset")
    dataset.generate_test_cases(my_pipeline) # Runs all inputs

    results = evals.evaluate(
    test_cases=dataset.test_cases,
    metrics=["answer_relevancy", "faithfulness"]
    )

    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import fallom from "@fallom/trace";

    fallom.evals.init({});

    async function myPipeline(messages: Array<{role: string, content: string}>) {
      const userQuery = messages[messages.length - 1].content;
      const response = await myRAGApp(userQuery);
      return { content: response.text };
    }

    const dataset = new fallom.evals.EvaluationDataset();
    await dataset.pull("my-dataset");
    await dataset.generateTestCases(myPipeline);  // Runs all inputs

    const results = await fallom.evals.evaluate({
      testCases: dataset.testCases,
      metrics: ["answer_relevancy", "faithfulness"]
    });
    ```
  </Tab>
</Tabs>

## Model Comparison

Compare how different models perform on the same dataset:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    from fallom import evals

    evals.init()

    comparison = evals.compare_models(
    dataset="my-dataset-key",
    models=["anthropic/claude-3.5-sonnet", "openai/gpt-4o", "google/gemini-2.0-flash"],
    metrics=["answer_relevancy", "faithfulness"],
    name="Model Comparison Q4 2024"
    )

    # Results show scores for each model:

    # - production (your original outputs)

    # - anthropic/claude-3.5-sonnet

    # - openai/gpt-4o

    # - google/gemini-2.0-flash

    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import fallom from "@fallom/trace";

    fallom.evals.init({});

    const comparison = await fallom.evals.compareModels({
      dataset: "my-dataset-key",
      models: ["anthropic/claude-3.5-sonnet", "openai/gpt-4o", "google/gemini-2.0-flash"],
      metrics: ["answer_relevancy", "faithfulness"],
      name: "Model Comparison Q4 2024"
    });
    ```
  </Tab>
</Tabs>

## Custom & Fine-Tuned Models

You can include your own fine-tuned or self-hosted models in comparisons:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    from fallom import evals

    evals.init()

    # Fine-tuned OpenAI model

    fine_tuned = evals.create_openai_model(
    "ft:gpt-4o-2024-08-06:my-org::abc123",
    name="my-fine-tuned"
    )

    # Self-hosted model (vLLM, Ollama, etc.)

    my_llama = evals.create_custom_model(
    name="my-llama-70b",
    endpoint="http://localhost:8000/v1/chat/completions",
    model_value="meta-llama/Llama-3.1-70B-Instruct"
    )

    # Compare all together

    comparison = evals.compare_models(
    dataset="my-dataset-key",
    models=[fine_tuned, my_llama, "openai/gpt-4o"],
    metrics=["answer_relevancy", "faithfulness"]
    )

    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    import fallom from "@fallom/trace";

    fallom.evals.init({});

    // Fine-tuned OpenAI model
    const fineTuned = fallom.evals.createOpenAIModel(
      "ft:gpt-4o-2024-08-06:my-org::abc123",
      { name: "my-fine-tuned" }
    );

    // Self-hosted model (vLLM, Ollama, etc.)
    const myLlama = fallom.evals.createCustomModel("my-llama-70b", {
      endpoint: "http://localhost:8000/v1/chat/completions",
      modelValue: "meta-llama/Llama-3.1-70B-Instruct"
    });

    // Compare all together
    const comparison = await fallom.evals.compareModels({
      dataset: "my-dataset-key",
      models: [fineTuned, myLlama, "openai/gpt-4o"],
      metrics: ["answer_relevancy", "faithfulness"]
    });
    ```
  </Tab>
</Tabs>

## Custom Judge Model

By default, evaluations use `openai/gpt-4o-mini` via OpenRouter as the judge. You can specify a different judge:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    results = evals.evaluate(
        dataset="my-dataset-key",
        metrics=["answer_relevancy"],
        judge_model="anthropic/claude-3.5-sonnet"  # Use Claude as judge
    )
    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    const results = await fallom.evals.evaluate({
      dataset: "my-dataset-key",
      metrics: ["answer_relevancy"],
      judgeModel: "anthropic/claude-3.5-sonnet"  // Use Claude as judge
    });
    ```
  </Tab>
</Tabs>

## API Reference

### `evaluate()`

Evaluate outputs against metrics. Use either `dataset` or `test_cases`.

<Tabs>
  <Tab title="Python">
    | Parameter     | Type                         | Default                | Description                         |
    | ------------- | ---------------------------- | ---------------------- | ----------------------------------- |
    | `dataset`     | `list[DatasetItem]` or `str` | —                      | Dataset items or Fallom dataset key |
    | `test_cases`  | `list[LLMTestCase]`          | —                      | Test cases from EvaluationDataset   |
    | `metrics`     | `list[str \| CustomMetric]`  | all built-in           | Metrics to run (built-in or custom) |
    | `judge_model` | `str`                        | `"openai/gpt-4o-mini"` | Model to use as judge               |
    | `name`        | `str`                        | auto-generated         | Name for this evaluation run        |
    | `description` | `str`                        | `None`                 | Optional description                |
    | `verbose`     | `bool`                       | `True`                 | Print progress                      |

    <Note>Either `dataset` or `test_cases` must be provided.</Note>
  </Tab>

  <Tab title="TypeScript">
    | Parameter     | Type                                | Default                | Description                         |
    | ------------- | ----------------------------------- | ---------------------- | ----------------------------------- |
    | `dataset`     | `DatasetItem[]` or `string`         | —                      | Dataset items or Fallom dataset key |
    | `testCases`   | `LLMTestCase[]`                     | —                      | Test cases from EvaluationDataset   |
    | `metrics`     | `Array<MetricName \| CustomMetric>` | all built-in           | Metrics to run (built-in or custom) |
    | `judgeModel`  | `string`                            | `"openai/gpt-4o-mini"` | Model to use as judge               |
    | `name`        | `string`                            | auto-generated         | Name for this evaluation run        |
    | `description` | `string`                            | `undefined`            | Optional description                |
    | `verbose`     | `boolean`                           | `true`                 | Print progress                      |

    <Note>Either `dataset` or `testCases` must be provided.</Note>
  </Tab>
</Tabs>

### `compare_models()` / `compareModels()`

Compare multiple models on the same dataset.

<Tabs>
  <Tab title="Python">
    | Parameter            | Type                         | Default                | Description                         |
    | -------------------- | ---------------------------- | ---------------------- | ----------------------------------- |
    | `dataset`            | `list[DatasetItem]` or `str` | required               | Dataset items or Fallom dataset key |
    | `models`             | `list[str or Model]`         | required               | Models to compare                   |
    | `metrics`            | `list[str \| CustomMetric]`  | all built-in           | Metrics to run (built-in or custom) |
    | `judge_model`        | `str`                        | `"openai/gpt-4o-mini"` | Model to use as judge               |
    | `include_production` | `bool`                       | `True`                 | Include original outputs            |
    | `name`               | `str`                        | auto-generated         | Name for this comparison run        |
  </Tab>

  <Tab title="TypeScript">
    | Parameter           | Type                                | Default                | Description                         |
    | ------------------- | ----------------------------------- | ---------------------- | ----------------------------------- |
    | `dataset`           | `DatasetItem[]` or `string`         | required               | Dataset items or Fallom dataset key |
    | `models`            | `Array<string \| Model>`            | required               | Models to compare                   |
    | `metrics`           | `Array<MetricName \| CustomMetric>` | all built-in           | Metrics to run (built-in or custom) |
    | `judgeModel`        | `string`                            | `"openai/gpt-4o-mini"` | Model to use as judge               |
    | `includeProduction` | `boolean`                           | `true`                 | Include original outputs            |
    | `name`              | `string`                            | auto-generated         | Name for this comparison run        |
  </Tab>
</Tabs>

### Model Helpers

| Function                                                     | Description                                        |
| ------------------------------------------------------------ | -------------------------------------------------- |
| `create_openai_model()` / `createOpenAIModel()`              | Create model for fine-tuned OpenAI or Azure OpenAI |
| `create_custom_model()` / `createCustomModel()`              | Create model for any OpenAI-compatible endpoint    |
| `create_model_from_callable()` / `createModelFromCallable()` | Create model from custom function                  |

### Metric Helpers

| Function / Class                     | Description                                   |
| ------------------------------------ | --------------------------------------------- |
| `custom_metric()` / `customMetric()` | Create a custom evaluation metric with G-Eval |
| `CustomMetric`                       | Class/interface for defining custom metrics   |

### EvaluationDataset

A class for managing datasets and test cases when using your own LLM pipeline.

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    dataset = evals.EvaluationDataset()

    # Methods

    dataset.pull(alias, version=None) # Pull dataset from Fallom
    dataset.add_golden(golden) # Add a golden record
    dataset.add_test_case(test_case) # Add a test case
    dataset.generate_test_cases(llm_app) # Auto-generate test cases
    dataset.clear_test_cases() # Clear all test cases

    # Properties

    dataset.goldens # List of Golden records
    dataset.test_cases # List of LLMTestCase records
    dataset.dataset_key # Fallom dataset key (if pulled)

    ```
  </Tab>

  <Tab title="TypeScript">
    ```typescript theme={null}
    const dataset = new fallom.evals.EvaluationDataset();

    // Methods
    await dataset.pull(alias, version?)              // Pull dataset from Fallom
    dataset.addGolden(golden)                        // Add a golden record
    dataset.addTestCase(testCase)                    // Add a test case
    await dataset.generateTestCases(llmApp)          // Auto-generate test cases
    dataset.clearTestCases()                         // Clear all test cases

    // Properties
    dataset.goldens      // Array of Golden records
    dataset.testCases    // Array of LLMTestCase records
    dataset.datasetKey   // Fallom dataset key (if pulled)
    ```
  </Tab>
</Tabs>

### LLMTestCase

A test case for evaluation containing input and actual output from your LLM.

<Tabs>
  <Tab title="Python">
    | Field             | Type                   | Description                    |
    | ----------------- | ---------------------- | ------------------------------ |
    | `input`           | `str`                  | The user input/query           |
    | `actual_output`   | `str`                  | Output from your LLM pipeline  |
    | `expected_output` | `str` (optional)       | Expected output for comparison |
    | `system_message`  | `str` (optional)       | System prompt used             |
    | `context`         | `list[str]` (optional) | Retrieved docs for RAG eval    |
    | `metadata`        | `dict` (optional)      | Additional metadata            |
  </Tab>

  <Tab title="TypeScript">
    | Field            | Type                  | Description                    |
    | ---------------- | --------------------- | ------------------------------ |
    | `input`          | `string`              | The user input/query           |
    | `actualOutput`   | `string`              | Output from your LLM pipeline  |
    | `expectedOutput` | `string` (optional)   | Expected output for comparison |
    | `systemMessage`  | `string` (optional)   | System prompt used             |
    | `context`        | `string[]` (optional) | Retrieved docs for RAG eval    |
    | `metadata`       | `object` (optional)   | Additional metadata            |
  </Tab>
</Tabs>

### Golden

A golden record from a dataset containing input and optionally expected output.

<Tabs>
  <Tab title="Python">
    | Field             | Type                   | Description                |
    | ----------------- | ---------------------- | -------------------------- |
    | `input`           | `str`                  | The user input/query       |
    | `expected_output` | `str` (optional)       | The expected/golden output |
    | `system_message`  | `str` (optional)       | System prompt              |
    | `context`         | `list[str]` (optional) | Context documents          |
    | `metadata`        | `dict` (optional)      | Additional metadata        |
  </Tab>

  <Tab title="TypeScript">
    | Field            | Type                  | Description                |
    | ---------------- | --------------------- | -------------------------- |
    | `input`          | `string`              | The user input/query       |
    | `expectedOutput` | `string` (optional)   | The expected/golden output |
    | `systemMessage`  | `string` (optional)   | System prompt              |
    | `context`        | `string[]` (optional) | Context documents          |
    | `metadata`       | `object` (optional)   | Additional metadata        |
  </Tab>
</Tabs>

## Next Steps

<CardGroup cols={2}>
  <Card title="Tracing" icon="route" href="/tracing">
    Log traces to enable dashboard evals
  </Card>

  <Card title="Model Testing" icon="flask" href="/model-testing">
    A/B test models in production
  </Card>

  <Card title="Prompt Testing" icon="pen-fancy" href="/prompt-testing">
    Version and test your prompts
  </Card>
</CardGroup>