Model A/B Testing

Run A/B tests on models with zero latency. Same session always gets same model (sticky assignment).

Create and manage your model configs in the dashboard.

Python
TypeScript

Setup

from fallom import models

models.init(api_key="your-fallom-api-key")

Basic Usage

from fallom import models

# Get assigned model for this session
model = models.get("summarizer-config", session_id)
# Returns: "gpt-4o" or "claude-3-5-sonnet" based on your config weights

agent = Agent(model=model)
agent.run(message)

Version Pinning

Pin to a specific config version, or use latest (default):

# Use latest version (default)
model = models.get("my-config", session_id)

# Pin to specific version
model = models.get("my-config", session_id, version=2)

Fallback for Resilience

Always provide a fallback so your app works even if Fallom is down:

model = models.get(
    "my-config",
    session_id,
    fallback="gpt-4o-mini"  # Used if config not found or Fallom unreachable
)

User Targeting

Override weighted distribution for specific users or segments:

model = models.get(
    "my-config",
    session_id,
    fallback="gpt-4o-mini",
    customer_id="user-123",  # For individual targeting
    context={                 # For rule-based targeting
        "plan": "enterprise",
        "region": "us-west"
    }
)

Setup

import { models } from "@fallom/trace";

models.init({ apiKey: "your-fallom-api-key" });

Basic Usage

import { models } from "@fallom/trace";

// Get assigned model for this session
const model = await models.get("summarizer-config", sessionId);
// Returns: "gpt-4o" or "claude-3-5-sonnet" based on your config weights

const response = await openai.chat.completions.create({ model, ... });

Fallback for Resilience

const model = await models.get("my-config", sessionId, {
  fallback: "gpt-4o-mini", // Used if config not found or Fallom unreachable
});

User Targeting

Override weighted distribution for specific users or segments:

const model = await models.get("my-config", sessionId, {
  fallback: "gpt-4o-mini",
  customerId: "user-123",           // For individual targeting
  context: {                         // For rule-based targeting
    plan: "enterprise",
    region: "us-west",
  },
});

Custom Model Providers

A/B test between standard models and custom-hosted models (Novita, Together, Fireworks, etc.):

import { models } from "@fallom/trace";
import { createOpenAI } from "@ai-sdk/openai";

// Get model from A/B test config
const modelId = await models.get("vision-model", sessionId, {
  fallback: "gpt-4o",
});

// Route to the correct provider
const model = createModelClient(modelId);

See Custom Model Providers below for the full setup.

User Targeting

Target specific users or segments to specific model variants. This is useful for:

Beta testing - Roll out new models to specific users first
Enterprise features - Give premium users access to better models
Gradual rollouts - Target by region, plan, or any custom attribute

How It Works

Targeting rules are evaluated client-side for zero latency:

Individual Targets - Exact match on customerId or any field
Rules - Condition-based targeting (all conditions in a rule must match)
Fallback - If no targeting matches, use weighted random distribution

Configuration

Configure targeting in the dashboard when editing a model config:

{
  "enabled": true,
  "individualTargets": [
    { "field": "customerId", "value": "vip-user-123", "variantIndex": 1 }
  ],
  "rules": [
    {
      "conditions": [
        { "field": "plan", "operator": "eq", "value": "enterprise" }
      ],
      "variantIndex": 1
    }
  ]
}

Supported Operators

Operator	Description	Example
`eq`	Equals	`plan = "enterprise"`
`neq`	Not equals	`plan ≠ "free"`
`in`	In list	`plan in ["enterprise", "business"]`
`nin`	Not in list	`region not in ["cn", "ru"]`
`contains`	Contains substring	`email contains "@acme.com"`
`startsWith`	Starts with	`region starts with "eu-"`
`endsWith`	Ends with	`email ends with ".gov"`

Custom Model Providers

A/B test between standard API models and custom-hosted models (self-hosted, Novita, Together, Fireworks, Ollama, etc.).

Dashboard Setup

Create a config with custom model names using any naming convention:

Variant	Model	Weight
Control	`gpt-4o`	50%
Custom	`custom:my-llama-70b`	50%

Use a prefix like custom:, together:, or local: to identify non-standard providers.

Provider Routing

Create a helper function to route model IDs to the correct provider:

TypeScript
Python

import { createOpenAI } from "@ai-sdk/openai";

function createModelClient(modelId: string) {
  // Custom-hosted models
  if (modelId.startsWith("custom:")) {
    return createOpenAI({
      apiKey: process.env.CUSTOM_API_KEY,
      baseURL: "https://your-custom-endpoint.com/v1",
    })(modelId.replace("custom:", ""));
  }

  // Together AI
  if (modelId.startsWith("together:")) {
    return createOpenAI({
      apiKey: process.env.TOGETHER_API_KEY,
      baseURL: "https://api.together.xyz/v1",
    })(modelId.replace("together:", ""));
  }

  // Default to OpenAI
  return createOpenAI()(modelId);
}

from openai import OpenAI

def create_model_client(model_id: str):
    # Custom-hosted models
    if model_id.startswith("custom:"):
        return OpenAI(
            api_key=os.environ["CUSTOM_API_KEY"],
            base_url="https://your-custom-endpoint.com/v1"
        ), model_id.replace("custom:", "")

    # Together AI
    if model_id.startswith("together:"):
        return OpenAI(
            api_key=os.environ["TOGETHER_API_KEY"],
            base_url="https://api.together.xyz/v1"
        ), model_id.replace("together:", "")

    # Default to OpenAI
    return OpenAI(), model_id

Full Example

TypeScript
Python

import { models } from "@fallom/trace";
import { createOpenAI } from "@ai-sdk/openai";
import { generateText } from "ai";

// Initialize once at startup
models.init({ apiKey: process.env.FALLOM_API_KEY });

async function chat(sessionId: string, message: string) {
  // Get A/B tested model
  const modelId = await models.get("my-agent", sessionId, {
    fallback: "gpt-4o-mini",
  });

  // Route to correct provider
  const model = createModelClient(modelId);

  // Use with Vercel AI SDK
  const result = await generateText({
    model,
    prompt: message,
  });

  return result.text;
}

from fallom import models

# Initialize once at startup
models.init(api_key=os.environ["FALLOM_API_KEY"])

def chat(session_id: str, message: str):
    # Get A/B tested model
    model_id = models.get("my-agent", session_id, fallback="gpt-4o-mini")

    # Route to correct provider
    client, model_name = create_model_client(model_id)

    # Use with OpenAI SDK
    response = client.chat.completions.create(
        model=model_name,
        messages=[{"role": "user", "content": message}]
    )

    return response.choices[0].message.content

Use Cases

Cost optimization - A/B test expensive vs cheap models
Latency testing - Compare self-hosted vs API latency
Gradual migration - Roll out new model providers safely
Fallback routing - Scale custom models to 0% instantly if issues arise

Resilience Guarantees

Zero Latency

Targeting evaluated client-side, no network call

Background Sync

Config sync never blocks your requests

Graceful Degradation

Returns fallback on any error

Sticky Sessions

Same session always gets same model

Next Steps

Prompt Management

Test different prompts alongside model variants.

View Analytics

Analyze experiment results in your dashboard.

Getting Started

Features

TypeScript Integrations

Python Integrations

AI Coding Tools

No-Code Integrations

Model A/B Testing

Setup

Basic Usage

Version Pinning

Fallback for Resilience

User Targeting

Setup

Basic Usage

Fallback for Resilience

User Targeting

Custom Model Providers

User Targeting

How It Works

Configuration

Supported Operators

Custom Model Providers

Dashboard Setup

Provider Routing

Full Example

Use Cases

Resilience Guarantees

Zero Latency

Background Sync

Graceful Degradation

Sticky Sessions

Next Steps

Prompt Management

View Analytics

​Setup

​Basic Usage

​Version Pinning

​Fallback for Resilience

​User Targeting

​Setup

​Basic Usage

​Fallback for Resilience

​User Targeting

​Custom Model Providers

​User Targeting

​How It Works

​Configuration

​Supported Operators

​Custom Model Providers

​Dashboard Setup

​Provider Routing

​Full Example

​Use Cases

​Resilience Guarantees

Zero Latency

Background Sync

Graceful Degradation

Sticky Sessions

​Next Steps

Prompt Management

View Analytics

Setup

Basic Usage

Version Pinning

Fallback for Resilience

User Targeting

Setup

Basic Usage

Fallback for Resilience

User Targeting

Custom Model Providers

User Targeting

How It Works

Configuration

Supported Operators

Custom Model Providers

Dashboard Setup

Provider Routing

Full Example

Use Cases

Resilience Guarantees

Next Steps