Language Models in SmythOS

This guide provides comprehensive information about configuring and using language models (LLMs) in the SmythOS SDK.

Overview

SmythOS supports a wide range of language model providers, from cloud-based services like OpenAI and Anthropic to locally-hosted models via Ollama. The SDK provides flexible configuration options to suit different use cases, from simple model selection to fine-grained parameter control.

Model Configuration Patterns

There are three main ways to configure models in SmythOS:

1. Simple String Notation

The quickest way to get started. Just specify the model name as a string:

const agent = new Agent({
    name: 'My Agent',
    behavior: 'You are a helpful assistant',
    model: 'gpt-4o', // Simple and straightforward
});
Copy

Benefits:

• Minimal configuration
• SDK keeps an up-to-date list of popular models
• Automatic provider detection

Common Model Names:

• OpenAI: 'gpt-4o', 'gpt-4-turbo', 'gpt-4', 'gpt-3.5-turbo'
• Anthropic: 'claude-4-sonnet', 'claude-3.5-sonnet', 'claude-3-opus', 'claude-3-sonnet', 'claude-3-haiku'
• Google: 'gemini-pro', 'gemini-1.5-pro', 'gemini-1.5-flash'
• DeepSeek: 'deepseek-chat', 'deepseek-coder'
• And many more...

2. Provider-Specific Simple Notation

When you need to specify a provider explicitly or use a model not in the default list:

import { Model } from '@smythos/sdk';

const agent = new Agent({
    name: 'My Agent',
    behavior: 'You are a helpful assistant',
    model: Model.OpenAI('gpt-4o'),
});
Copy

Benefits:

• Explicit provider selection
• Works with custom or new models
• Type-safe provider interface

3. Advanced Configuration

For full control over model behavior and parameters:

import { Model } from '@smythos/sdk';

const agent = new Agent({
    name: 'My Agent',
    behavior: 'You are a helpful assistant',
    model: Model.OpenAI('gpt-4o', {
        temperature: 0.7, // Control randomness (0.0 - 2.0)
        maxTokens: 2000, // Maximum response length
        topP: 0.9, // Nucleus sampling parameter
        frequencyPenalty: 0.0, // Reduce repetition (0.0 - 2.0)
        presencePenalty: 0.0, // Encourage topic diversity (0.0 - 2.0)
    }),
});
Copy

Benefits:

• Fine-tuned control over model behavior
• Performance optimization
• Cost management via token limits

Available Providers

SmythOS supports the following LLM providers:

OpenAI

Access GPT-4, GPT-3.5, and other OpenAI models:

model: Model.OpenAI('gpt-4o', {
    temperature: 0.7,
    maxTokens: 4000,
    topP: 1.0,
    frequencyPenalty: 0.0,
    presencePenalty: 0.0,
});
Copy

Popular Models:

• gpt-4o - Latest optimized GPT-4 (multimodal)
• gpt-4-turbo - Fast GPT-4 variant
• gpt-4 - Standard GPT-4
• gpt-3.5-turbo - Fast and cost-effective

API Key: Store in vault as "openai": "sk-..." (see API Key Management below).

Anthropic (Claude)

Access Claude models from Anthropic:

model: Model.Anthropic('claude-4-sonnet', {
    temperature: 1.0,
    maxTokens: 8192,
    topP: 0.9,
});
Copy

Popular Models:

• claude-4-sonnet - Latest Claude model
• claude-3.5-sonnet - Balanced performance
• claude-3-opus - Most capable model
• claude-3-sonnet - Fast and balanced
• claude-3-haiku - Fastest, most cost-effective

API Key: Store in vault as "anthropic": "sk-ant-..." (see API Key Management below).

GoogleAI (Gemini)

Access Google's Gemini models:

model: Model.GoogleAI('gemini-1.5-pro', {
    temperature: 0.8,
    maxTokens: 2048,
    topP: 0.95,
});
Copy

Popular Models:

• gemini-1.5-pro - Advanced reasoning and long context
• gemini-1.5-flash - Fast and efficient
• gemini-pro - Standard model

API Key: Store in vault as "googleai": "..." (see API Key Management below).

Groq

Ultra-fast inference with Groq's LPU™ technology:

model: Model.Groq('llama-3.1-70b-versatile', {
    temperature: 0.5,
    maxTokens: 1024,
});
Copy

Popular Models:

• llama-3.1-70b-versatile - Meta's Llama 3.1 (70B)
• llama-3.1-8b-instant - Smaller, faster Llama 3.1
• mixtral-8x7b-32768 - Mixtral MoE model

API Key: Store in vault as "groq": "gsk_..." (see API Key Management below).

Ollama (Local Models)

Run models locally with Ollama:

model: Model.Ollama('llama3.2', {
    temperature: 0.7,
    numCtx: 4096, // Context window size
});
Copy

Setup:

1. Install Ollama from ollama.ai
2. Pull models: ollama pull llama3.2
3. Start Ollama server (usually runs automatically)

Popular Models:

• llama3.2 - Latest Llama model
• mistral - Mistral 7B
• codellama - Code-specialized Llama
• phi3 - Microsoft's Phi-3

Configuration: No API key needed for local Ollama. Configure host URL in SRE config if not using default http://localhost:11434

DeepSeek

Access DeepSeek models:

model: Model.DeepSeek('deepseek-chat', {
    temperature: 0.7,
    maxTokens: 2000,
});
Copy

Popular Models:

• deepseek-chat - General conversation model
• deepseek-coder - Specialized for coding tasks

API Key: Store in vault as "deepseek": "..." (see API Key Management below).

TogetherAI

Access various open-source models via TogetherAI:

model: Model.TogetherAI('meta-llama/Llama-3-70b-chat-hf', {
    temperature: 0.7,
    maxTokens: 2000,
});
Copy

API Key: Store in vault as "togetherai": "..." (see API Key Management below).

xAI (Grok)

model: Model.xAI('grok-beta', {
    temperature: 0.7,
    maxTokens: 2000,
});
Copy

API Key: Store in vault as "xai": "..." (see API Key Management below).

Perplexity

model: Model.Perplexity('llama-3.1-sonar-large-128k-online', {
    temperature: 0.7,
    maxTokens: 2000,
});
Copy

API Key: Store in vault as "perplexity": "..." (see API Key Management below).

Common Model Parameters

Most providers support these common parameters. Here's a quick reference table:

Parameter	Type	Range/Format	Description	Supported By
temperature	number	0.0 - 2.0	Controls randomness in responses. Lower = more deterministic, higher = more creative	All providers
maxTokens	number	1 - model limit	Maximum number of tokens in the response	All providers
topP	number	0.0 - 1.0	Nucleus sampling parameter. Controls diversity via cumulative probability	Most providers
topK	number	0 - ∞	Limits token selection to top K most likely tokens. 0 = disabled	Ollama, some providers
frequencyPenalty	number	0.0 - 2.0	Reduces repetition of token sequences	OpenAI, compatible providers
presencePenalty	number	0.0 - 2.0	Encourages talking about new topics	OpenAI, compatible providers
stopSequences	string[]	Array of strings	Sequences where the model will stop generating	Most providers
inputTokens	number	1 - model limit	Maximum context window size (input tokens). Should be ≤ model's official context window	All providers
outputTokens	number	1 - model limit	The maximum tokens that the model can generate in a single response	All providers
maxThinkingTokens	number	1 - model limit	Maximum tokens for reasoning/thinking (reasoning models only)	OpenAI o1, compatible models
baseURL	string	Valid URL	Custom API endpoint URL for model inference	Most providers

Detailed Parameter Descriptions

temperature

Controls randomness in responses (typically 0.0 - 2.0):

• 0.0 - Deterministic, focused responses
• 0.5 - Balanced creativity and consistency
• 1.0 - More creative and varied
• 2.0 - Highly random and creative

model: Model.OpenAI('gpt-4o', { temperature: 0.7 });
Copy

maxTokens

Maximum number of tokens in the response:

model: Model.OpenAI('gpt-4o', { maxTokens: 2000 });
Copy

Note: Tokens are not the same as words. Roughly 1 token ≈ 0.75 words.

topP

Nucleus sampling parameter (0.0 - 1.0):

• Controls diversity via cumulative probability
• Lower values = more focused, higher values = more diverse

model: Model.Anthropic('claude-3-sonnet', { topP: 0.9 });
Copy

frequencyPenalty

Reduces repetition of token sequences (0.0 - 2.0):

• Higher values discourage repeating the same phrases

model: Model.OpenAI('gpt-4o', { frequencyPenalty: 0.5 });
Copy

presencePenalty

Encourages talking about new topics (0.0 - 2.0):

• Higher values encourage introducing new topics

model: Model.OpenAI('gpt-4o', { presencePenalty: 0.5 });
Copy

API Key Management

SmythOS stores API keys securely in a Vault system. By default, the SDK uses a JSON file-based vault located at .smyth/vault.json.

Default Vault (JSON File)

The SDK automatically initializes with a JSON vault at .smyth/vault.json. This file stores API keys for different providers:

{
    "default": {
        "openai": "sk-...",
        "anthropic": "sk-ant-...",
        "googleai": "...",
        "groq": "gsk_...",
        "deepseek": "...",
        "togetherai": "...",
        "xai": "...",
        "perplexity": "..."
    }
}
Copy

Location Options:

• Project-level: .smyth/vault.json in your project directory (recommended)
• User-level: ~/.smyth/vault.json in your home directory (applies to all projects)

Environment Variables (Optional)

You can reference environment variables within the vault file using the $env() syntax:

{
    "default": {
        "openai": "$env(OPENAI_API_KEY)",
        "anthropic": "$env(ANTHROPIC_API_KEY)",
        "googleai": "$env(GOOGLE_API_KEY)"
    }
}
Copy

Then set your environment variables:

export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GOOGLE_API_KEY="..."
Copy

Custom Vault Configuration

For advanced use cases, you can configure a different vault system:

import { SRE } from '@smythos/sdk/core';

SRE.init({
    Vault: {
        Connector: 'JSONFileVault',
        Settings: {
            file: './custom-vault.json',
        },
    },
});
Copy

Or use cloud-based secret management systems like AWS Secrets Manager for production environments.

Team-Based API Keys

The vault supports multiple teams with isolated API keys:

{
    "default": {
        "openai": "sk-default-key..."
    },
    "team-production": {
        "openai": "sk-prod-key...",
        "anthropic": "sk-ant-prod-..."
    },
    "team-development": {
        "openai": "sk-dev-key..."
    }
}
Copy

Agents automatically use keys from their assigned team's vault section.

Best Practices

1. Choose the Right Model for Your Use Case

• Complex reasoning: GPT-4, Claude Opus, Gemini Pro
• Fast responses: GPT-3.5 Turbo, Claude Haiku, Gemini Flash
• Cost-effective: GPT-3.5 Turbo, Groq models, Claude Haiku
• Privacy-sensitive: Ollama (local models)
• Long context: Claude models (100k+ tokens), Gemini 1.5 Pro

2. Tune Temperature Based on Task

• Factual tasks (data extraction, analysis): temperature: 0.0 - 0.3
• Balanced tasks (chatbots, assistants): temperature: 0.5 - 0.7
• Creative tasks (writing, brainstorming): temperature: 0.8 - 1.2

3. Set Appropriate Token Limits

• Set maxTokens to prevent unexpectedly long (and costly) responses
• Consider the model's context window when setting limits
• Remember: input + output tokens count toward the limit

4. Test with Different Providers

Different providers excel at different tasks. Test multiple options to find the best fit for your specific use case.

5. Monitor Costs

• Use cheaper models for development/testing
• Set conservative maxTokens limits
• Consider caching responses for repeated queries
• Use streaming to provide partial responses quickly

Examples

Example 1: Cost-Optimized Configuration

import { Agent, Model } from '@smythos/sdk';

const agent = new Agent({
    name: 'Budget Assistant',
    behavior: 'You are a helpful assistant',
    // Use GPT-3.5 for cost efficiency
    model: Model.OpenAI('gpt-3.5-turbo', {
        temperature: 0.5,
        maxTokens: 500, // Keep responses concise
    }),
});
Copy

Example 2: High-Quality Reasoning

const agent = new Agent({
    name: 'Research Assistant',
    behavior: 'You are an expert researcher',
    // Use Claude Opus for complex reasoning
    model: Model.Anthropic('claude-3-opus', {
        temperature: 0.3, // More focused
        maxTokens: 4096, // Allow detailed responses
    }),
});
Copy

Example 3: Local Development

const agent = new Agent({
    name: 'Dev Assistant',
    behavior: 'You are a coding assistant',
    // Use Ollama for local development
    model: Model.Ollama('llama3.2', {
        temperature: 0.7,
        numCtx: 4096,
    }),
});
Copy

Example 4: Fast Responses with Groq

const agent = new Agent({
    name: 'Quick Responder',
    behavior: 'You provide quick answers',
    // Use Groq for ultra-fast inference
    model: Model.Groq('llama-3.1-70b-versatile', {
        temperature: 0.5,
        maxTokens: 1024,
    }),
});
Copy

Example 5: Model Override in .smyth Import

import { Agent, Model } from '@smythos/sdk';
import path from 'path';

const agentPath = path.resolve(__dirname, './workflow-agent.smyth');

// Import workflow but override the model
const agent = Agent.import(agentPath, {
    model: Model.Anthropic('claude-3.5-sonnet', {
        temperature: 0.8,
    }),
});
Copy

Next Steps

Now that you understand model configuration, explore:

• Building Agents with Skills - Learn how to add capabilities to your agents
• Streaming Responses - Handle real-time model outputs
• Advanced Topics - Deep dive into advanced features