Polar Llama

pip install polar-llama==0.3.0

Development Installation

import polars as pl
from polar_llama import string_to_message, inference_async, Provider
import dotenv

# Load environment variables
dotenv.load_dotenv()

# Create a DataFrame with questions
questions = [
    'What is the capital of France?',
    'What is the difference between polars and pandas?',
    'Explain async programming in Python'
]

df = pl.DataFrame({'Questions': questions})

# Convert questions to LLM messages
df = df.with_columns(
    prompt=string_to_message("Questions", message_type='user')
)

# Run parallel inference (default model is now gpt-4o-mini)
df = df.with_columns(
    answer=inference_async('prompt', provider=Provider.OPENAI,
                          model='gpt-4o-mini')
)

# Display results
print(df)

import polars as pl
from polar_llama import (
    Provider, mcp_tools, tools_to_response_model,
    execute_tool_calls, tool_results_to_message,
    combine_messages, inference_messages,
)

# 1. Get tool definitions - from an MCP server, or plain dicts / Pydantic models
tools = mcp_tools("http://localhost:8811/mcp")

# 2. Emission: the LLM parameterizes zero or more calls per row.
#    This is ordinary structured output - nothing is executed.
ToolCalls = tools_to_response_model(tools)

df = pl.DataFrame({"meal": [
    "two eggs, sourdough toast with butter, black coffee",
    "chipotle chicken bowl, no rice, extra guac",
]})

df = df.with_columns(
    calls=pl.col("meal").llama.inference_async(
        provider=Provider.OPENAI, model="gpt-4o-mini",
        response_model=ToolCalls,
    )
)

# 3. Execution: all calls across all rows run concurrently on the Rust
#    async runtime. Arguments are validated against each tool's own input
#    schema before any network call; invalid calls fail fast as data.
df = df.with_columns(
    results=execute_tool_calls(
        pl.col("calls"),
        transport="http://localhost:8811/mcp",
        tools=tools,           # optional: enables pre-execution validation
        concurrency=64,
        timeout_s=30,
    )
)

# 4. Synthesis: fold results back through a second inference pass.
df = df.with_columns(
    answer=inference_messages(
        combine_messages(
            pl.col("meal").llama.to_message(role="user"),
            tool_results_to_message(pl.col("results")),
        ),
        provider=Provider.OPENAI, model="gpt-4o-mini",
    )
)

Non-MCP targets

Anything callable from Python can be a tool target via the executor escape hatch — a database, an internal API, a local function. Runs on a thread pool with the same errors-as-data semantics.

Full example: calorie tracker

Three explicit turns, each a column: meal text -> emit searches -> execute in parallel -> synthesize summary

df.with_columns(
    response=inference_async(
        pl.col("prompt"),
        provider=Provider.ANTHROPIC,
        model="claude-opus-4-8",
        system_prompt="You are a helpful assistant.",  # required for caching to help
        cache=True,
    )
)

Advanced: fine-grained CacheConfig

import polars as pl
from polar_llama import Predict, Signature, BootstrapFewShot, InstructionOptimizer, evaluate

# 1. Declare the task
module = Predict(
    Signature("question -> answer", instructions="Answer concisely."),
    provider="openai",
    model="gpt-4o-mini",
)

# 2. Labeled training data
trainset = pl.DataFrame({
    "question": ["What is 2+2?", "Capital of France?", "Largest planet?"],
    "answer": ["4", "Paris", "Jupiter"],
})

# 3. A metric: (gold row, prediction) -> bool | float
def exact_match(example, prediction):
    return example["answer"].strip().lower() == (prediction["answer"] or "").strip().lower()

# 4a. Bootstrap few-shot demos from rows the model already gets right
compiled = BootstrapFewShot(metric=exact_match, max_demos=4).compile(module, trainset)

# 4b. Or search for better instructions (COPRO-style)
optimizer = InstructionOptimizer(metric=exact_match, n_candidates=4)
compiled = optimizer.compile(module, trainset)
print(optimizer.history)  # [(instructions, score), ...]

# 5. Run the optimized module on new data - outputs land in pred_* columns
result = compiled(pl.DataFrame({"question": ["What is 3+3?"]}))
print(result["pred_answer"])

# Score any module against a labeled set
print(evaluate(compiled, trainset, exact_match).score)

Typed output fields

Output fields can be typed and described for stronger structured outputs

import polars as pl
from polar_llama import string_to_message, inference_async

# Create a DataFrame with system prompts and user questions
df = pl.DataFrame({
    "system_prompt": [
        "You are a helpful assistant.",
        "You are a math expert.",
        "You are a creative writer."
    ],
    "user_question": [
        "What's the weather like today?",
        "Solve x^2 + 5x + 6 = 0",
        "Write a haiku about coding"
    ]
})

# Convert both columns to messages
df = df.with_columns([
    string_to_message("system_prompt", message_type="system").alias("system_message"),
    string_to_message("user_question", message_type="user").alias("user_message")
])

# Combine messages into conversations
from polar_llama import combine_messages, inference_messages
df = df.with_columns(
    combine_messages("system_message", "user_message").alias("conversation")
)

# Run inference with combined messages
df = df.with_columns(
    inference_messages("conversation",
           provider="openai",
           model="gpt-4o-mini").alias("response")
)

print(df.select(["user_question", "response"]))

import polars as pl
from polar_llama import string_to_message, inference_async, Provider

# Load customer feedback data
feedback_df = pl.DataFrame({
    'customer_id': [101, 102, 103, 104, 105],
    'feedback': [
        'The product is amazing but shipping was slow',
        'Great quality, highly recommend!',
        'Disappointed with customer service',
        'Perfect for my needs, will buy again',
        'Product arrived damaged, requesting refund'
    ]
})

# Create sentiment analysis prompts
sentiment_prompt = """Analyze the sentiment of this customer feedback
and classify it as Positive, Negative, or Neutral.
Also provide a brief reason.

Feedback: {feedback}"""

df = feedback_df.with_columns(
    prompt=pl.format(sentiment_prompt, pl.col('feedback'))
)

# Convert to messages and run inference
df = df.with_columns(
    message=string_to_message("prompt", message_type='user')
)

df = df.with_columns(
    sentiment_analysis=inference_async('message',
                                      provider=Provider.OPENAI,
                                      model='gpt-4o-mini')
)

# Extract key insights
print(df.select(['customer_id', 'feedback', 'sentiment_analysis']))

df = df.with_columns(
    answer=inference_async('prompt',
                          provider=Provider.OPENAI,
                          model='gpt-4o-mini')  # was gpt-4-turbo before 0.3.0
)

Requests no longer hardcode temperature / max_tokens — the o-series and GPT-5 reject those parameters. Pricing data and o200k tokenizer detection added for GPT-5/4.1/o-series. Supports OPENAI_BASE_URL for proxies and gateways.

df = df.with_columns(
    answer=inference_async('prompt',
                          provider=Provider.ANTHROPIC,
                          model='claude-opus-4-8')  # claude-3-opus-20240229 was retired
)

Pricing data added for Claude 4.x / Fable 5. Supports cache=True prompt caching and ANTHROPIC_BASE_URL for proxies and gateways.

df = df.with_columns(
    answer=inference_async('prompt',
                          provider=Provider.GEMINI,
                          model='gemini-2.5-flash')  # was gemini-1.5-pro
)

Gemini now has native system_instruction support and native JSON-schema structured outputs (response_json_schema). Structured-output requests now authenticate via the x-goog-api-key header instead of the previously-broken OpenAI-style Bearer auth.

df = df.with_columns(
    answer=inference_async('prompt',
                          provider=Provider.GROQ,
                          model='llama-3.3-70b-versatile')  # llama3-70b-8192 was decommissioned
)

Requests no longer hardcode temperature / max_tokens.

# Requires AWS credentials configured
df = df.with_columns(
    answer=inference_async('prompt',
                          provider='bedrock',
                          # was anthropic.claude-3-haiku-20240307-v1:0
                          model='us.anthropic.claude-haiku-4-5-20251001-v1:0')
)

Bedrock now works from the synchronous inference expression (previously errored), structured outputs route through the AWS SDK instead of a raw HTTP POST, region now respects AWS_REGION / AWS_DEFAULT_REGION before falling back to us-east-1, and prompt caching emits a real CachePoint block (5-minute cache only). Pricing data added for Bedrock Claude 4.5.

• Fixed: Gemini structured outputs previously sent OpenAI-style Bearer auth and always failed; now authenticates via x-goog-api-key.
• Fixed: Bedrock structured outputs previously attempted a raw HTTP POST; now route through the AWS SDK like plain requests. Bedrock also now works from the synchronous inference expression.
• Security: TLS verification is no longer disabled — the shared client uses rustls with the OS certificate store. Fixed RUSTSEC-2025-0020 (pyo3 buffer overflow) by upgrading pyo3 0.23 → 0.27.
• Fixed: Windows wheels build again — the AWS SDK now uses thering rustls provider instead of aws-lc-rs, whose native build failed under MSVC.
• Performance: Single shared HTTP client with connection pooling, bounded request concurrency via buffered streams instead of unbounded join_all, JSON schemas compiled once per batch, and roughly 400 lines of duplicated per-row dispatch removed from the expression layer.
• Changed: Removed deprecated Rust constructors new() / with_model() (deprecated since 0.2.0) — use new_with_model(). Removed import-time debug printing from the Python package and native module.
• Full details in CHANGELOG.md under the 0.3.0 entry.