Self Host LLMGateway

LLMGateway is a self-hostable platform that provides a unified API gateway for multiple LLM providers. This guide offers two simple options to get started.

Prerequisites

• Latest Docker
• API keys for the LLM providers you want to use (OpenAI, Anthropic, etc.)

Option 1: Unified Docker Image (Simplest)

This option uses a single Docker container that includes all services (UI, API, Gateway, Database, Redis).

# Run the container
docker run -d \
  --name llmgateway \
  --restart unless-stopped \
  -p 3002:3002 \
  -p 3003:3003 \
  -p 3005:3005 \
  -p 4001:4001 \
  -p 4002:4002 \
  -v ~/llmgateway_data:/var/lib/postgresql/data \
  -e AUTH_SECRET=your-secret-key-here \
  ghcr.io/theopenco/llmgateway-unified:latest

Note: it is recommended to use the latest version tag from here instead of latest: https://github.com/theopenco/llmgateway/releases

Using Docker Compose (Alternative for unified image)

# Download the compose file
curl -O https://raw.githubusercontent.com/theopenco/llmgateway/main/infra/docker-compose.unified.yml
curl -O https://raw.githubusercontent.com/theopenco/llmgateway/main/.env.example

# Configure environment
cp .env.example .env
# Edit .env with your configuration

# Start the service
docker compose -f docker-compose.unified.yml up -d

Note: it is recommended to replace the latest version tag in the image with the latest version from here: https://github.com/theopenco/llmgateway/releases

Option 2: Separate Services with Docker Compose

This option uses separate containers for each service, offering more flexibility.

# Clone the repository
git clone https://github.com/theopenco/llmgateway.git
cd llmgateway

# Configure environment
cp .env.example .env
# Edit .env with your configuration

# Start the services
docker compose -f infra/docker-compose.split.yml up -d

Note: it is recommended to replace the latest version tag in all images in the compose file with the latest version from here: https://github.com/theopenco/llmgateway/releases

Accessing Your LLMGateway

After starting either option, you can access:

• Web Interface: http://localhost:3002
• Documentation: http://localhost:3005
• API Endpoint: http://localhost:4002
• Gateway Endpoint: http://localhost:4001

Required Configuration

At minimum, you need to set these environment variables:

# Database (change the password!)
POSTGRES_PASSWORD=your_secure_password_here

# Authentication
AUTH_SECRET=your-secret-key-here

# LLM Provider API Keys (add the ones you need)
LLM_OPENAI_API_KEY=sk-...
LLM_ANTHROPIC_API_KEY=sk-ant-...

Basic Management Commands

For Unified Docker (Option 1)

# View logs
docker logs llmgateway

# Restart container
docker restart llmgateway

# Stop container
docker stop llmgateway

For Docker Compose (Option 2)

# View logs
docker compose -f infra/docker-compose.split.yml logs -f

# Restart services
docker compose -f infra/docker-compose.split.yml restart

# Stop services
docker compose -f infra/docker-compose.split.yml down

Build locally

To build locally, you can use the *.local.yml compose file in the infra directory, which will build the images from the source code.

All provider API keys

You can set any of the following API keys:

LLM_OPENAI_API_KEY=
LLM_ANTHROPIC_API_KEY=

Multiple API Keys and Load Balancing

LLMGateway supports multiple API keys per provider for load balancing and increased availability. Simply provide comma-separated values for your API keys:

# Multiple OpenAI keys for load balancing
LLM_OPENAI_API_KEY=sk-key1,sk-key2,sk-key3

# Multiple Anthropic keys
LLM_ANTHROPIC_API_KEY=sk-ant-key1,sk-ant-key2

Health-Aware Routing

The gateway automatically tracks the health of each API key and routes requests to healthy keys. If a key experiences consecutive errors, it will be temporarily skipped. Keys that return authentication errors (401/403) are permanently blacklisted until restart.

Related Configuration Values

For providers that require additional configuration (like Google Vertex), you can specify multiple values that correspond to each API key. The gateway will always use the matching index:

# Multiple Google Vertex configurations
LLM_GOOGLE_VERTEX_API_KEY=key1,key2,key3
LLM_GOOGLE_CLOUD_PROJECT=project-a,project-b,project-c
LLM_GOOGLE_VERTEX_REGION=us-central1,europe-west1,asia-east1

When the gateway selects key2, it will automatically use project-b and europe-west1. If you have fewer configuration values than keys, the last value will be reused for remaining keys.

Next Steps

Once your LLMGateway is running:

1. Open the web interface at http://localhost:3002
2. Create your first organization and project
3. Generate API keys for your applications
4. Test the gateway by making API calls to http://localhost:4001