The LLM (Large Language Model) subsystem provides access to various language models from different providers. It supports text generation, chat completions, embeddings, and model management with unified interfaces across providers.
Important: LLM connectors work differently from other SRE connectors. They do not have constructor settings. Instead, configuration is managed through:
ModelsProvider (typically JSONModelsProvider using models.json)Role: Development and testing LLM connector
Summary: Provides a simple echo service that returns the input prompt. Useful for testing workflows without consuming API credits or requiring external services.
Configuration: No constructor settings. Configured through models.json.
Example Model Entry:
{
"echo-test": {
"label": "Echo Test Model",
"llm": "Echo",
"credentials": "None"
}
}
Use Cases:
Role: OpenAI API connector (GPT-3.5, GPT-4, etc.)
Summary: Provides access to OpenAI's language models including GPT-3.5-turbo, GPT-4, and embedding models with full feature support.
Configuration: No constructor settings. Requires API key in vault and model configuration.
Required Vault Entries:
openai:apiKey - OpenAI API keyExample Model Entry:
{
"gpt-4": {
"label": "GPT-4",
"llm": "OpenAI",
"modelId": "gpt-4",
"credentials": "Internal",
"baseURL": "https://api.openai.com/v1",
"params": {
"temperature": 0.7,
"max_tokens": 4096
}
}
}
Use Cases:
Role: DeepSeek API connector
Summary: Provides access to DeepSeek's language models using OpenAI-compatible API interface. Uses the OpenAI connector internally.
Configuration: No constructor settings. Uses OpenAI connector with custom baseURL.
Required Vault Entries:
deepseek:apiKey - DeepSeek API keyExample Model Entry:
{
"deepseek-chat": {
"label": "DeepSeek Chat",
"llm": "DeepSeek",
"modelId": "deepseek-chat",
"credentials": "Internal",
"baseURL": "https://api.deepseek.com/v1"
}
}
Role: Google AI (Gemini) connector
Summary: Provides access to Google's Gemini models with support for multimodal inputs, function calling, and Google's advanced AI capabilities.
Configuration: No constructor settings.
Required Vault Entries:
googleai:apiKey - Google AI API keyExample Model Entry:
{
"gemini-pro": {
"label": "Gemini Pro",
"llm": "GoogleAI",
"modelId": "gemini-pro",
"credentials": "Internal"
}
}
Role: Anthropic Claude connector
Summary: Provides access to Anthropic's Claude models with strong performance in reasoning, analysis, and safe AI interactions.
Configuration: No constructor settings.
Required Vault Entries:
anthropic:apiKey - Anthropic API keyExample Model Entry:
{
"claude-3-sonnet": {
"label": "Claude 3 Sonnet",
"llm": "Anthropic",
"modelId": "claude-3-sonnet-20240229",
"credentials": "Internal"
}
}
Role: Groq high-speed inference connector
Summary: Provides ultra-fast inference using Groq's specialized hardware. Supports various open-source models with exceptional speed performance.
Configuration: No constructor settings.
Required Vault Entries:
groq:apiKey - Groq API keyExample Model Entry:
{
"llama-3-70b": {
"label": "Llama 3 70B",
"llm": "Groq",
"modelId": "llama3-70b-8192",
"credentials": "Internal"
}
}
Role: Amazon Bedrock connector
Summary: Provides access to various foundation models through AWS Bedrock service with enterprise security and compliance features.
Configuration: No constructor settings.
Required Vault Entries:
aws:accessKeyId - AWS access key IDaws:secretAccessKey - AWS secret access keyaws:region - AWS regionExample Model Entry:
{
"claude-bedrock": {
"label": "Claude 3 on Bedrock",
"llm": "Bedrock",
"modelId": "anthropic.claude-3-sonnet-20240229-v1:0",
"credentials": "BedrockVault",
"region": "us-east-1"
}
}
Models are configured through the ModelsProvider (typically JSONModelsProvider):
{
"model-name": {
"label": "Human readable name",
"llm": "ConnectorName",
"modelId": "actual-model-id",
"credentials": "CredentialType",
"baseURL": "api-endpoint",
"params": {
"temperature": 0.7,
"max_tokens": 4096
}
}
}
| Type | Description |
|---|---|
"None" |
No credentials required (Echo connector) |
"Internal" |
API key from vault (most providers) |
"BedrockVault" |
AWS credentials for Bedrock |
"VertexAIVault" |
Google Cloud credentials for Vertex AI |
The JSONModelsProvider connector has constructor settings:
| Setting | Type | Required | Default | Description |
|---|---|---|---|---|
models |
string | TLLMModelsList | No | Built-in models | Directory path to models.json or models object |
mode |
string | No | "merge" |
How to handle custom models: "merge" or "replace" |
Example Configuration:
ModelsProvider: {
Connector: 'JSONModelsProvider',
Settings: {
models: './config/models.json',
mode: 'merge'
}
}
Credentials are stored in the vault using this format:
{provider}:apiKey - For most providers (e.g., openai:apiKey)aws:accessKeyId, aws:secretAccessKey, aws:region - For AWS services// Store credentials in vault
await vault.set('openai:apiKey', 'sk-...');
await vault.set('anthropic:apiKey', 'sk-ant-...');
await vault.set('groq:apiKey', 'gsk_...');
import { LLM } from '@smythos/sdk';
const llm = new LLM();
// Model name references models.json entry
const response = await llm.chat({
model: 'gpt-4',
messages: [{ role: 'user', content: 'Hello!' }],
});
// In models.json
{
"custom-gpt": {
"label": "Custom GPT-4",
"llm": "OpenAI",
"modelId": "gpt-4",
"credentials": "Internal",
"params": {
"temperature": 0.1,
"max_tokens": 8192
}
}
}
// Usage
const response = await llm.chat({
model: 'custom-gpt',
messages: [{ role: 'user', content: 'Hello!' }]
});