Skip to Content
DocumentationFeaturesLarge Language Models (LLMs)

Large Language Models (LLMs)

Alith provides seamless integration with various Large Language Models (LLMs), allowing you to easily switch between models like GPT-4, GPT-3.5, Claude, DeepSeek and others. Below, you’ll find examples of how to initialize and use LLMs in Rust, Python, and Node.js.

OpenAI Models

Set the API key.

  • Unix
export OPENAI_API_KEY=<your API key>
  • Windows
$env:OPENAI_API_KEY = "<your API key>"

Write the code.

use alith::{Agent, Chat, LLM};
 
#[tokio::main]
async fn main() -> Result<(), anyhow::Error> {
    let model = LLM::from_model_name("gpt-4")?;
    let agent = Agent::new("simple agent", model)
        .preamble("You are a comedian here to entertain the user using humour and jokes.");
 
    let response = agent.prompt("Entertain me!").await?;
 
    println!("{}", response);
 
    Ok(())
}

OpenAI API Compatible Models

Here, we take the DeepSeek model as the example.

use alith::{Agent, Chat, LLM};
 
#[tokio::main]
async fn main() -> Result<(), anyhow::Error> {
    let model = LLM::openai_compatible_model(
        "<Your API Key>", // Replace with your api key or read it from env.
        "api.deepseek.com",
        "deepseek-chat", // or `deepseek-reasoner` for DeepSeek R1 Model
    )?;
    let agent = Agent::new("simple agent", model)
        .preamble("You are a comedian here to entertain the user using humour and jokes.");
 
    let response = agent.prompt("Entertain me!").await?;
 
    println!("{}", response);
 
    Ok(())
}

Anthropic Models

Set the API key.

  • Unix
export ANTHROPIC_API_KEY=<your API key>
  • Windows
$env:ANTHROPIC_API_KEY = "<your API key>"

Write the code.

use alith::{Agent, Chat, LLM};
 
#[tokio::main]
async fn main() -> Result<(), anyhow::Error> {
    let model = LLM::from_model_name("claude-3-5-sonnet")?;
    let agent = Agent::new("simple agent", model)
        .preamble("You are a comedian here to entertain the user using humour and jokes.");
 
    let response = agent.prompt("Entertain me!").await?;
 
    println!("{}", response);
 
    Ok(())
}

HuggingFace Models

use alith::HuggingFaceLoader;
 
fn main() -> Result<(), anyhow::Error> {
    let _path = HuggingFaceLoader::new().load_file("model.safetensors", "gpt2")?;
    Ok(())
}

Note: we can use the HF_ENDPOINT env to set different huggingface endpoints.

GGUF Models

use alith::{GgufLoader, GgufLoaderTrait};
 
fn main() -> Result<(), anyhow::Error> {
    let _model = GgufLoader::default()
        .hf_quant_file_url("https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/blob/main/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf")
        .load()?;
 
    // By default we attempt to extract everything we need from the GGUF file.
    // If you need to specifiy the tokenizer or chat template to use, you can add a hf repo to load from.
    let _model = GgufLoader::default()
        .hf_quant_file_url("https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/blob/main/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf")
        .hf_config_repo_id("meta-llama/Meta-Llama-3-8B-Instruct")
        .load()?;
 
    // We can also load the model from the local path
    let _model = GgufLoader::default()
        .local_quant_file_path("/root/models/qwen2.5-1.5b-instruct-q5_k_m.gguf")
        .load()?;
    Ok(())
}
Last updated on
Get StartedTools