> ## Documentation Index
> Fetch the complete documentation index at: https://docs.fallom.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Model A/B Testing
> Test different LLM models to optimize performance, quality, and cost
Run A/B tests on models with zero latency. Same session always gets same model (sticky assignment).
Create and manage your model configs in the [dashboard](https://app.fallom.com).
## Setup
```python theme={null}
from fallom import models
models.init(api_key="your-fallom-api-key")
```
## Basic Usage
```python theme={null}
from fallom import models
# Get assigned model for this session
model = models.get("summarizer-config", session_id)
# Returns: "gpt-4o" or "claude-3-5-sonnet" based on your config weights
agent = Agent(model=model)
agent.run(message)
```
### Version Pinning
Pin to a specific config version, or use latest (default):
```python theme={null}
# Use latest version (default)
model = models.get("my-config", session_id)
# Pin to specific version
model = models.get("my-config", session_id, version=2)
```
### Fallback for Resilience
Always provide a fallback so your app works even if Fallom is down:
```python theme={null}
model = models.get(
"my-config",
session_id,
fallback="gpt-4o-mini" # Used if config not found or Fallom unreachable
)
```
### User Targeting
Override weighted distribution for specific users or segments:
```python theme={null}
model = models.get(
"my-config",
session_id,
fallback="gpt-4o-mini",
customer_id="user-123", # For individual targeting
context={ # For rule-based targeting
"plan": "enterprise",
"region": "us-west"
}
)
```
## Setup
```typescript theme={null}
import { models } from "@fallom/trace";
models.init({ apiKey: "your-fallom-api-key" });
```
## Basic Usage
```typescript theme={null}
import { models } from "@fallom/trace";
// Get assigned model for this session
const model = await models.get("summarizer-config", sessionId);
// Returns: "gpt-4o" or "claude-3-5-sonnet" based on your config weights
const response = await openai.chat.completions.create({ model, ... });
```
### Fallback for Resilience
```typescript theme={null}
const model = await models.get("my-config", sessionId, {
fallback: "gpt-4o-mini", // Used if config not found or Fallom unreachable
});
```
### User Targeting
Override weighted distribution for specific users or segments:
```typescript theme={null}
const model = await models.get("my-config", sessionId, {
fallback: "gpt-4o-mini",
customerId: "user-123", // For individual targeting
context: { // For rule-based targeting
plan: "enterprise",
region: "us-west",
},
});
```
### Custom Model Providers
A/B test between standard models and custom-hosted models (Novita, Together, Fireworks, etc.):
```typescript theme={null}
import { models } from "@fallom/trace";
import { createOpenAI } from "@ai-sdk/openai";
// Get model from A/B test config
const modelId = await models.get("vision-model", sessionId, {
fallback: "gpt-4o",
});
// Route to the correct provider
const model = createModelClient(modelId);
```
See [Custom Model Providers](#custom-model-providers-1) below for the full setup.
## User Targeting
Target specific users or segments to specific model variants. This is useful for:
* **Beta testing** - Roll out new models to specific users first
* **Enterprise features** - Give premium users access to better models
* **Gradual rollouts** - Target by region, plan, or any custom attribute
### How It Works
Targeting rules are evaluated client-side for zero latency:
1. **Individual Targets** - Exact match on `customerId` or any field
2. **Rules** - Condition-based targeting (all conditions in a rule must match)
3. **Fallback** - If no targeting matches, use weighted random distribution
### Configuration
Configure targeting in the dashboard when editing a model config:
```json theme={null}
{
"enabled": true,
"individualTargets": [
{ "field": "customerId", "value": "vip-user-123", "variantIndex": 1 }
],
"rules": [
{
"conditions": [
{ "field": "plan", "operator": "eq", "value": "enterprise" }
],
"variantIndex": 1
}
]
}
```
### Supported Operators
| Operator | Description | Example |
| ------------ | ------------------ | ------------------------------------ |
| `eq` | Equals | `plan = "enterprise"` |
| `neq` | Not equals | `plan ≠ "free"` |
| `in` | In list | `plan in ["enterprise", "business"]` |
| `nin` | Not in list | `region not in ["cn", "ru"]` |
| `contains` | Contains substring | `email contains "@acme.com"` |
| `startsWith` | Starts with | `region starts with "eu-"` |
| `endsWith` | Ends with | `email ends with ".gov"` |
## Custom Model Providers
A/B test between standard API models and custom-hosted models (self-hosted, Novita, Together, Fireworks, Ollama, etc.).
### Dashboard Setup
Create a config with custom model names using any naming convention:
| Variant | Model | Weight |
| ------- | --------------------- | ------ |
| Control | `gpt-4o` | 50% |
| Custom | `custom:my-llama-70b` | 50% |
Use a prefix like `custom:`, `together:`, or `local:` to identify non-standard providers.
### Provider Routing
Create a helper function to route model IDs to the correct provider:
```typescript theme={null}
import { createOpenAI } from "@ai-sdk/openai";
function createModelClient(modelId: string) {
// Custom-hosted models
if (modelId.startsWith("custom:")) {
return createOpenAI({
apiKey: process.env.CUSTOM_API_KEY,
baseURL: "https://your-custom-endpoint.com/v1",
})(modelId.replace("custom:", ""));
}
// Together AI
if (modelId.startsWith("together:")) {
return createOpenAI({
apiKey: process.env.TOGETHER_API_KEY,
baseURL: "https://api.together.xyz/v1",
})(modelId.replace("together:", ""));
}
// Default to OpenAI
return createOpenAI()(modelId);
}
```
```python theme={null}
from openai import OpenAI
def create_model_client(model_id: str):
# Custom-hosted models
if model_id.startswith("custom:"):
return OpenAI(
api_key=os.environ["CUSTOM_API_KEY"],
base_url="https://your-custom-endpoint.com/v1"
), model_id.replace("custom:", "")
# Together AI
if model_id.startswith("together:"):
return OpenAI(
api_key=os.environ["TOGETHER_API_KEY"],
base_url="https://api.together.xyz/v1"
), model_id.replace("together:", "")
# Default to OpenAI
return OpenAI(), model_id
```
### Full Example
```typescript theme={null}
import { models } from "@fallom/trace";
import { createOpenAI } from "@ai-sdk/openai";
import { generateText } from "ai";
// Initialize once at startup
models.init({ apiKey: process.env.FALLOM_API_KEY });
async function chat(sessionId: string, message: string) {
// Get A/B tested model
const modelId = await models.get("my-agent", sessionId, {
fallback: "gpt-4o-mini",
});
// Route to correct provider
const model = createModelClient(modelId);
// Use with Vercel AI SDK
const result = await generateText({
model,
prompt: message,
});
return result.text;
}
```
```python theme={null}
from fallom import models
# Initialize once at startup
models.init(api_key=os.environ["FALLOM_API_KEY"])
def chat(session_id: str, message: str):
# Get A/B tested model
model_id = models.get("my-agent", session_id, fallback="gpt-4o-mini")
# Route to correct provider
client, model_name = create_model_client(model_id)
# Use with OpenAI SDK
response = client.chat.completions.create(
model=model_name,
messages=[{"role": "user", "content": message}]
)
return response.choices[0].message.content
```
### Use Cases
* **Cost optimization** - A/B test expensive vs cheap models
* **Latency testing** - Compare self-hosted vs API latency
* **Gradual migration** - Roll out new model providers safely
* **Fallback routing** - Scale custom models to 0% instantly if issues arise
## Resilience Guarantees
Targeting evaluated client-side, no network call
Config sync never blocks your requests
Returns fallback on any error
Same session always gets same model
## Next Steps
Test different prompts alongside model variants.
Analyze experiment results in your dashboard.