Open source library

Polar Llama

A Python library for parallel LLM inference across providers, built on Polars DataFrames.

v0.2.0Released January 15, 2025

Version0.5.2latest 0.5.1 0.5.0 0.3.0 0.2.2 0.2.1 0.2.0 0.1.7

You are viewing documentation for version 0.2.0. A newer version (0.5.2) is available.

Overview

Polar Llama is a Python library that enables parallel inference calls to multiple Large Language Model providers through Polars dataframes. It streamlines batch processing of AI queries without serial request delays, making it ideal for data-intensive AI applications.

⚡Concurrent Processing

Send multiple inference requests in parallel without waiting for individual completions

🐻Polars Integration

Leverages efficient Polars dataframe operations for request management

💬Multi-turn Conversations

Supports context-preserving conversations across multiple message exchanges

🔌Multiple Providers

Connects with OpenAI, Anthropic, Gemini, Groq, and AWS Bedrock models

🎯Structured Output Support — New in 0.2.0

Native support for structured outputs with Pydantic models and JSON schema validation

Installation

Using pip

bash

pip install polar-llama==0.2.0

Development Installation

bash

maturin develop

Quick Start

Get started with a simple example

python

import polars as pl
from polar_llama import string_to_message, inference_async, Provider
import dotenv

# Load environment variables
dotenv.load_dotenv()

# Create a DataFrame with questions
questions = [
    'What is the capital of France?',
    'What is the difference between polars and pandas?',
    'Explain async programming in Python'
]

df = pl.DataFrame({'Questions': questions})

# Convert questions to LLM messages
df = df.with_columns(
    prompt=string_to_message("Questions", message_type='user')
)

# Run parallel inference
df = df.with_columns(
    answer=inference_async('prompt', provider=Provider.OPENAI,
                          model='gpt-4o-mini')
)

# Display results
print(df)

Structured Outputs

New in 0.2.0

What are Structured Outputs?

Get type-safe, validated responses from LLMs in a predictable format

Structured outputs allow you to define the exact schema you want the LLM to follow, ensuring responses are properly formatted and can be directly used in your data pipelines. This is perfect for extracting specific information, generating consistent data, or integrating LLM outputs with databases and APIs.

✅Type Safety

Define your output schema with Pydantic models for guaranteed type correctness

🎯Validation

Automatic validation ensures responses match your schema before processing

🔄Consistency

Get predictable, parseable outputs across all your inference requests

⚙️Easy Integration

Seamlessly integrate with databases, APIs, and data processing pipelines

Basic Structured Output

Define a simple schema and get structured responses

python

import polars as pl
from polar_llama import string_to_message, inference_async, Provider
from pydantic import BaseModel

# Define your output schema
class ProductInfo(BaseModel):
    name: str
    price: float
    category: str
    in_stock: bool

# Create prompts
prompts = [
    "Extract product info: iPhone 15 Pro for $999 in Electronics, available",
    "Extract product info: Nike Air Max shoes for $129.99 in Footwear, sold out",
    "Extract product info: Laptop Stand for $49.99 in Accessories, in stock"
]

df = pl.DataFrame({'prompt': prompts})

# Convert to messages
df = df.with_columns(
    message=string_to_message("prompt", message_type='user')
)

# Run inference with structured output
df = df.with_columns(
    product=inference_async(
        'message',
        provider=Provider.OPENAI,
        model='gpt-4o-2024-08-06',
        response_model=ProductInfo  # Specify your Pydantic model
    )
)

# Access structured fields directly
print(df.select(['product']))

Complex Structured Output

Handle nested schemas and lists for complex data extraction

python

import polars as pl
from polar_llama import string_to_message, inference_async, Provider
from pydantic import BaseModel, Field
from typing import List

# Define nested schema
class Person(BaseModel):
    name: str
    role: str
    email: str

class Meeting(BaseModel):
    title: str
    date: str
    duration_minutes: int
    attendees: List[Person]
    action_items: List[str]
    priority: str = Field(description="high, medium, or low")

# Meeting transcripts
transcripts = [
    """Team standup on Nov 10, 2025, 30 minutes.
    Attended by John (PM, john@example.com) and Sarah (Dev, sarah@example.com).
    Action items: Review PR #123, Update documentation.
    Priority: high""",
]

df = pl.DataFrame({'transcript': transcripts})

# Prepare prompts
df = df.with_columns(
    prompt=pl.format(
        "Extract structured meeting information from this transcript: {}",
        pl.col('transcript')
    )
)

# Convert to messages
df = df.with_columns(
    message=string_to_message("prompt", message_type='user')
)

# Get structured meeting data
df = df.with_columns(
    meeting=inference_async(
        'message',
        provider=Provider.OPENAI,
        model='gpt-4o-2024-08-06',
        response_model=Meeting
    )
)

# Access nested fields
print(df.select(['meeting']))

Sentiment Analysis with Structured Outputs

Extract sentiment, confidence, and key themes in a structured format

python

import polars as pl
from polar_llama import string_to_message, inference_async, Provider
from pydantic import BaseModel, Field
from typing import List, Literal

class SentimentAnalysis(BaseModel):
    sentiment: Literal["positive", "negative", "neutral"]
    confidence: float = Field(ge=0.0, le=1.0, description="Confidence score 0-1")
    key_themes: List[str] = Field(description="Main themes in the text")
    suggested_action: str = Field(description="Recommended follow-up action")

# Customer feedback
feedback = [
    "The product quality is excellent, but delivery took 2 weeks. Customer service was helpful.",
    "Terrible experience. Product broke after one day and no response to my emails.",
    "Works as expected. Nothing special but does the job."
]

df = pl.DataFrame({'feedback': feedback})

# Create analysis prompts
df = df.with_columns(
    prompt=pl.format(
        "Analyze this customer feedback: {}",
        pl.col('feedback')
    )
)

# Convert to messages
df = df.with_columns(
    message=string_to_message("prompt", message_type='user')
)

# Get structured sentiment analysis
df = df.with_columns(
    analysis=inference_async(
        'message',
        provider=Provider.ANTHROPIC,
        model='claude-3-haiku-20240307',
        response_model=SentimentAnalysis
    )
)

# Results are now fully typed and validated
print(df.select(['feedback', 'analysis']))

Structured Data Extraction

Extract and normalize data from unstructured text at scale

python

import polars as pl
from polar_llama import string_to_message, inference_async, Provider
from pydantic import BaseModel, Field
from typing import Optional
from datetime import date

class Invoice(BaseModel):
    invoice_number: str
    vendor_name: str
    invoice_date: str
    due_date: str
    total_amount: float
    currency: str
    tax_amount: Optional[float]
    payment_terms: Optional[str]

# Unstructured invoice text
invoices_text = [
    "Invoice #INV-2024-001 from Acme Corp dated Jan 15, 2024. Total: $1,250.00. Due in 30 days.",
    "INVOICE 2024-ABC-999 - Widget Co - Date: 2024-02-01 - Amount: £850.50 (inc. £141.75 VAT) - Net 15",
]

df = pl.DataFrame({'raw_text': invoices_text})

# Create extraction prompts
df = df.with_columns(
    prompt=pl.format(
        "Extract invoice details from: {}",
        pl.col('raw_text')
    )
)

# Convert to messages
df = df.with_columns(
    message=string_to_message("prompt", message_type='user')
)

# Extract structured invoice data
df = df.with_columns(
    invoice=inference_async(
        'message',
        provider=Provider.OPENAI,
        model='gpt-4o-2024-08-06',
        response_model=Invoice
    )
)

# Now you have clean, structured invoice data
print(df.select(['invoice']))

Examples & Cookbooks

Multi-Message Conversations

Maintain context across multiple messages for more natural interactions

python

import polars as pl
from polar_llama import string_to_message, inference_async

# Create a DataFrame with system prompts and user questions
df = pl.DataFrame({
    "system_prompt": [
        "You are a helpful assistant.",
        "You are a math expert.",
        "You are a creative writer."
    ],
    "user_question": [
        "What's the weather like today?",
        "Solve x^2 + 5x + 6 = 0",
        "Write a haiku about coding"
    ]
})

# Convert both columns to messages
df = df.with_columns([
    string_to_message("system_prompt", message_type="system").alias("system_message"),
    string_to_message("user_question", message_type="user").alias("user_message")
])

# Combine messages into conversations
from polar_llama import combine_messages, inference_messages
df = df.with_columns(
    combine_messages("system_message", "user_message").alias("conversation")
)

# Run inference with combined messages
df = df.with_columns(
    inference_messages("conversation",
           provider="openai",
           model="gpt-4").alias("response")
)

print(df.select(["user_question", "response"]))

Data Analysis Pipeline

Process customer feedback at scale

python

import polars as pl
from polar_llama import string_to_message, inference_async, Provider

# Load customer feedback data
feedback_df = pl.DataFrame({
    'customer_id': [101, 102, 103, 104, 105],
    'feedback': [
        'The product is amazing but shipping was slow',
        'Great quality, highly recommend!',
        'Disappointed with customer service',
        'Perfect for my needs, will buy again',
        'Product arrived damaged, requesting refund'
    ]
})

# Create sentiment analysis prompts
sentiment_prompt = """Analyze the sentiment of this customer feedback
and classify it as Positive, Negative, or Neutral.
Also provide a brief reason.

Feedback: {feedback}"""

df = feedback_df.with_columns(
    prompt=pl.format(sentiment_prompt, pl.col('feedback'))
)

# Convert to messages and run inference
df = df.with_columns(
    message=string_to_message("prompt", message_type='user')
)

df = df.with_columns(
    sentiment_analysis=inference_async('message',
                                      provider=Provider.OPENAI,
                                      model='gpt-4o-mini')
)

# Extract key insights
print(df.select(['customer_id', 'feedback', 'sentiment_analysis']))

Batch Content Generation

Generate multiple content variations in parallel

python

import polars as pl
from polar_llama import string_to_message, inference_async, Provider

# Product data
products = pl.DataFrame({
    'product_name': ['Wireless Headphones', 'Smart Watch', 'Laptop Stand'],
    'features': [
        'noise-canceling, 30-hour battery, comfortable',
        'fitness tracking, heart rate monitor, waterproof',
        'adjustable height, aluminum build, cable management'
    ],
    'target_audience': ['music lovers', 'fitness enthusiasts', 'remote workers']
})

# Create dynamic prompts for each product
prompt_template = """Create a compelling product description for:
Product: {product_name}
Features: {features}
Target Audience: {target_audience}

Write in a marketing tone, 2-3 sentences."""

df = products.with_columns(
    prompt=pl.format(prompt_template,
                    pl.col('product_name'),
                    pl.col('features'),
                    pl.col('target_audience'))
)

# Convert to messages
df = df.with_columns(
    message=string_to_message("prompt", message_type='user')
)

# Generate descriptions in parallel
df = df.with_columns(
    description=inference_async('message',
                               provider=Provider.ANTHROPIC,
                               model='claude-3-haiku-20240307')
)

print(df.select(['product_name', 'description']))

Code Review Assistant

Automated code review suggestions

python

import polars as pl
from polar_llama import string_to_message, inference_async, Provider

# Code snippets to review
code_samples = pl.DataFrame({
    'file_name': ['auth.py', 'database.py', 'api.py'],
    'code_snippet': [
        '''
def authenticate(password):
    if password == "admin123":
        return True
    return False
        ''',
        '''
def get_user(user_id):
    query = f"SELECT * FROM users WHERE id = {user_id}"
    return db.execute(query)
        ''',
        '''
def process_data(data):
    result = []
    for item in data:
        result.append(item * 2)
    return result
        '''
    ]
})

# Create review prompts
review_template = """Review this code for:
1. Security issues
2. Performance problems
3. Best practice violations

Code:
{code}

Provide specific suggestions for improvement."""

df = code_samples.with_columns(
    prompt=pl.format(review_template, pl.col('code_snippet'))
)

# Get AI code reviews
df = df.with_columns(
    message=string_to_message("prompt", message_type='user')
)

df = df.with_columns(
    review=inference_async('message',
                          provider=Provider.OPENAI,
                          model='gpt-4')
)

print(df.select(['file_name', 'review']))

Provider Support

Polar Llama supports multiple LLM providers

OpenAI

python

df = df.with_columns(
    answer=inference_async('prompt',
                          provider=Provider.OPENAI,
                          model='gpt-4o-mini')
)

Structured OutputsSupported on gpt-4o-2024-08-06 and later models with response_model parameter

Anthropic (Claude)

python

df = df.with_columns(
    answer=inference_async('prompt',
                          provider=Provider.ANTHROPIC,
                          model='claude-3-5-sonnet-20241022')
)

Structured OutputsSupported on Claude 3.5 Sonnet and later with response_model parameter

AWS Bedrock

python

# Requires AWS credentials configured
df = df.with_columns(
    answer=inference_async('prompt',
                          provider='bedrock',
                          model='anthropic.claude-3-haiku-20240307-v1:0')
)

Google Gemini

python

df = df.with_columns(
    answer=inference_async('prompt',
                          provider=Provider.GEMINI,
                          model='gemini-pro')
)

Groq

python

df = df.with_columns(
    answer=inference_async('prompt',
                          provider=Provider.GROQ,
                          model='mixtral-8x7b-32768')
)

Advanced Features

New in 0.2.0

Structured Output Configuration

Configure structured outputs with custom validation and error handling:

python

from pydantic import BaseModel, Field, validator

class CustomOutput(BaseModel):
    value: int = Field(ge=0, le=100, description="Value between 0-100")
    category: str

    @validator('category')
    def validate_category(cls, v):
        allowed = ['A', 'B', 'C']
        if v not in allowed:
            raise ValueError(f'Category must be one of {allowed}')
        return v

# Use in inference
df = df.with_columns(
    result=inference_async(
        'message',
        provider=Provider.OPENAI,
        model='gpt-4o-2024-08-06',
        response_model=CustomOutput,
        max_retries=3  # Retry on validation failures
    )
)

Environment Configuration

Set up your API keys in a .env file:

.envbash

OPENAI_API_KEY=your_openai_key
ANTHROPIC_API_KEY=your_anthropic_key
GEMINI_API_KEY=your_gemini_key
GROQ_API_KEY=your_groq_key
AWS_ACCESS_KEY_ID=your_aws_key
AWS_SECRET_ACCESS_KEY=your_aws_secret
AWS_REGION=us-east-1

Testing

Run tests with configured providers:

bash

pip install -r tests/requirements.txt
pytest tests/ -v
cargo test --test model_client_tests -- --nocapture

Upcoming Features

Streaming response capabilities with structured outputs
Additional provider integrations
Enhanced validation and error recovery

Common Use Cases

📊Data Analysis

Process large datasets with AI insights - sentiment analysis, classification, entity extraction with validated structured outputs

✍️Content Generation

Generate product descriptions, marketing copy, or documentation at scale with consistent formatting

🔍Research & Summarization

Summarize documents, extract key points with structured metadata, or answer questions about large text corpora

🤖Automation

Automate repetitive AI tasks like code review, email categorization, or data enrichment with type-safe outputs

Resources

→GitHub Repository →PyPI Package →Polars Documentation

Licensed under MIT.

Questions or issues? Open one on GitHub.