Open source library

Polar Llama

A Python library for parallel LLM inference across providers, built on Polars DataFrames.

v0.1.7Released November 8, 2024
View on GitHubPyPI package
Version0.5.2latest0.5.10.5.00.3.00.2.20.2.10.2.00.1.7
You are viewing documentation for version 0.1.7. A newer version (0.5.2) is available.

Overview

Polar Llama is a Python library that enables parallel inference calls to multiple Large Language Model providers through Polars dataframes. It streamlines batch processing of AI queries without serial request delays, making it ideal for data-intensive AI applications.

Concurrent Processing

Send multiple inference requests in parallel without waiting for individual completions

🐻Polars Integration

Leverages efficient Polars dataframe operations for request management

💬Multi-turn Conversations

Supports context-preserving conversations across multiple message exchanges

🔌Multiple Providers

Connects with OpenAI, Anthropic, Gemini, Groq, and AWS Bedrock models

Installation

Using pip

bash
pip install polar-llama==0.1.7

Development Installation

bash
maturin develop

Quick Start

Get started with a simple example

python
import polars as pl
from polar_llama import string_to_message, inference_async, Provider
import dotenv

# Load environment variables
dotenv.load_dotenv()

# Create a DataFrame with questions
questions = [
    'What is the capital of France?',
    'What is the difference between polars and pandas?',
    'Explain async programming in Python'
]

df = pl.DataFrame({'Questions': questions})

# Convert questions to LLM messages
df = df.with_columns(
    prompt=string_to_message("Questions", message_type='user')
)

# Run parallel inference
df = df.with_columns(
    answer=inference_async('prompt', provider=Provider.OPENAI,
                          model='gpt-4o-mini')
)

# Display results
print(df)

Examples & Cookbooks

Multi-Message Conversations

Maintain context across multiple messages for more natural interactions

python
import polars as pl
from polar_llama import string_to_message, inference_async

# Create a DataFrame with system prompts and user questions
df = pl.DataFrame({
    "system_prompt": [
        "You are a helpful assistant.",
        "You are a math expert.",
        "You are a creative writer."
    ],
    "user_question": [
        "What's the weather like today?",
        "Solve x^2 + 5x + 6 = 0",
        "Write a haiku about coding"
    ]
})

# Convert both columns to messages
df = df.with_columns([
    string_to_message("system_prompt", message_type="system").alias("system_message"),
    string_to_message("user_question", message_type="user").alias("user_message")
])

# Combine messages into conversations
from polar_llama import combine_messages, inference_messages
df = df.with_columns(
    combine_messages("system_message", "user_message").alias("conversation")
)

# Run inference with combined messages
df = df.with_columns(
    inference_messages("conversation",
           provider="openai",
           model="gpt-4").alias("response")
)

print(df.select(["user_question", "response"]))

Data Analysis Pipeline

Process customer feedback at scale

python
import polars as pl
from polar_llama import string_to_message, inference_async, Provider

# Load customer feedback data
feedback_df = pl.DataFrame({
    'customer_id': [101, 102, 103, 104, 105],
    'feedback': [
        'The product is amazing but shipping was slow',
        'Great quality, highly recommend!',
        'Disappointed with customer service',
        'Perfect for my needs, will buy again',
        'Product arrived damaged, requesting refund'
    ]
})

# Create sentiment analysis prompts
sentiment_prompt = """Analyze the sentiment of this customer feedback
and classify it as Positive, Negative, or Neutral.
Also provide a brief reason.

Feedback: {feedback}"""

df = feedback_df.with_columns(
    prompt=pl.format(sentiment_prompt, pl.col('feedback'))
)

# Convert to messages and run inference
df = df.with_columns(
    message=string_to_message("prompt", message_type='user')
)

df = df.with_columns(
    sentiment_analysis=inference_async('message',
                                      provider=Provider.OPENAI,
                                      model='gpt-4o-mini')
)

# Extract key insights
print(df.select(['customer_id', 'feedback', 'sentiment_analysis']))

Batch Content Generation

Generate multiple content variations in parallel

python
import polars as pl
from polar_llama import string_to_message, inference_async, Provider

# Product data
products = pl.DataFrame({
    'product_name': ['Wireless Headphones', 'Smart Watch', 'Laptop Stand'],
    'features': [
        'noise-canceling, 30-hour battery, comfortable',
        'fitness tracking, heart rate monitor, waterproof',
        'adjustable height, aluminum build, cable management'
    ],
    'target_audience': ['music lovers', 'fitness enthusiasts', 'remote workers']
})

# Create dynamic prompts for each product
prompt_template = """Create a compelling product description for:
Product: {product_name}
Features: {features}
Target Audience: {target_audience}

Write in a marketing tone, 2-3 sentences."""

df = products.with_columns(
    prompt=pl.format(prompt_template,
                    pl.col('product_name'),
                    pl.col('features'),
                    pl.col('target_audience'))
)

# Convert to messages
df = df.with_columns(
    message=string_to_message("prompt", message_type='user')
)

# Generate descriptions in parallel
df = df.with_columns(
    description=inference_async('message',
                               provider=Provider.ANTHROPIC,
                               model='claude-3-haiku-20240307')
)

print(df.select(['product_name', 'description']))

Code Review Assistant

Automated code review suggestions

python
import polars as pl
from polar_llama import string_to_message, inference_async, Provider

# Code snippets to review
code_samples = pl.DataFrame({
    'file_name': ['auth.py', 'database.py', 'api.py'],
    'code_snippet': [
        '''
def authenticate(password):
    if password == "admin123":
        return True
    return False
        ''',
        '''
def get_user(user_id):
    query = f"SELECT * FROM users WHERE id = {user_id}"
    return db.execute(query)
        ''',
        '''
def process_data(data):
    result = []
    for item in data:
        result.append(item * 2)
    return result
        '''
    ]
})

# Create review prompts
review_template = """Review this code for:
1. Security issues
2. Performance problems
3. Best practice violations

Code:
{code}

Provide specific suggestions for improvement."""

df = code_samples.with_columns(
    prompt=pl.format(review_template, pl.col('code_snippet'))
)

# Get AI code reviews
df = df.with_columns(
    message=string_to_message("prompt", message_type='user')
)

df = df.with_columns(
    review=inference_async('message',
                          provider=Provider.OPENAI,
                          model='gpt-4')
)

print(df.select(['file_name', 'review']))

Provider Support

Polar Llama supports multiple LLM providers

OpenAI

python
df = df.with_columns(
    answer=inference_async('prompt',
                          provider=Provider.OPENAI,
                          model='gpt-4o-mini')
)

Anthropic (Claude)

python
df = df.with_columns(
    answer=inference_async('prompt',
                          provider=Provider.ANTHROPIC,
                          model='claude-3-5-sonnet-20241022')
)

AWS Bedrock

python
# Requires AWS credentials configured
df = df.with_columns(
    answer=inference_async('prompt',
                          provider='bedrock',
                          model='anthropic.claude-3-haiku-20240307-v1:0')
)

Google Gemini

python
df = df.with_columns(
    answer=inference_async('prompt',
                          provider=Provider.GEMINI,
                          model='gemini-pro')
)

Groq

python
df = df.with_columns(
    answer=inference_async('prompt',
                          provider=Provider.GROQ,
                          model='mixtral-8x7b-32768')
)

Advanced Features

Environment Configuration

Set up your API keys in a .env file:

.envbash
OPENAI_API_KEY=your_openai_key
ANTHROPIC_API_KEY=your_anthropic_key
GEMINI_API_KEY=your_gemini_key
GROQ_API_KEY=your_groq_key
AWS_ACCESS_KEY_ID=your_aws_key
AWS_SECRET_ACCESS_KEY=your_aws_secret
AWS_REGION=us-east-1

Testing

Run tests with configured providers:

bash
pip install -r tests/requirements.txt
pytest tests/ -v
cargo test --test model_client_tests -- --nocapture

Upcoming Features

  • Function calling support for structured outputs
  • Streaming response capabilities
  • Additional provider integrations

Common Use Cases

📊Data Analysis

Process large datasets with AI insights — sentiment analysis, classification, entity extraction

✍️Content Generation

Generate product descriptions, marketing copy, or documentation at scale

🔍Research & Summarization

Summarize documents, extract key points, or answer questions about large text corpora

🤖Automation

Automate repetitive AI tasks like code review, email categorization, or data enrichment

Resources
GitHub RepositoryPyPI PackagePolars Documentation

Licensed under MIT.

Questions or issues? Open one on GitHub.