Why I Use Go for AI Backends (Not Python)

Every AI product I've shipped runs a Go backend. Not Python. Python is the default for AI — and it's the wrong default for backend services. Here's why I choose Go, where Python still wins, and the specific patterns that make Go excellent for AI-orchestrating backends.

The common assumption

"AI = Python." This is true for:

• Model training (PyTorch, JAX)
• Local inference (transformers, vLLM)
• Data processing (pandas, NumPy)
• Research notebooks

It's not true for:

• API servers that call LLM APIs
• Orchestration services that coordinate AI pipelines
• Webhook handlers that process events and dispatch to LLMs
• Business logic that happens alongside AI inference

Most "AI backends" don't run models — they call APIs. The AI computation happens at OpenAI, Anthropic, or your vLLM server. Your backend is a coordinator, not an inference engine.

Go's advantages for AI-orchestrating backends

Concurrency model

LLM API calls block for 300ms–3 seconds. During that time, your backend needs to handle other requests. In Python (without async): the entire server thread is blocked. In Go: the goroutine blocks, but the scheduler runs other goroutines.

// Go: 10,000 concurrent requests handling LLM calls
// each goroutine is cheap (~2KB stack vs ~1MB thread)
func handleRequest(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    
    // This blocks the goroutine, not the server
    response, err := llm.Complete(ctx, prompt)
    if err != nil {
        http.Error(w, err.Error(), 500)
        return
    }
    json.NewEncoder(w).Encode(response)
}

A t3.medium Go server handles ~5,000 concurrent LLM requests without breaking a sweat. A comparable Python (Flask/Django) server handles ~50 before performance degrades.

Memory efficiency

Python's GIL, object overhead, and garbage collection make it memory-hungry for concurrent workloads. A Go service handling 1,000 concurrent LLM calls uses ~100MB RAM. The Python equivalent: ~1–3GB.

At scale, this matters directly: fewer EC2 instances, smaller Fargate tasks, lower monthly bills.

Deployment simplicity

Go compiles to a single static binary:

FROM golang:1.22-alpine AS builder
WORKDIR /app
COPY . .
RUN go build -o api ./cmd/api

FROM scratch  # literally empty base image
COPY --from=builder /app/api /api
CMD ["/api"]

The resulting Docker image: ~15MB. Python equivalents: 300MB–1GB (with all dependencies). Smaller images = faster cold starts, cheaper storage, faster CI.

Type safety catches bugs before production

Go's static typing prevents an entire class of runtime errors:

type LLMResponse struct {
    Content    string  `json:"content"`
    TokensUsed int     `json:"tokens_used"`
    Model      string  `json:"model"`
}

func parseResponse(body []byte) (LLMResponse, error) {
    var resp LLMResponse
    if err := json.Unmarshal(body, &resp); err != nil {
        return LLMResponse{}, fmt.Errorf("parse: %w", err)
    }
    return resp, nil
}

Python's dynamic typing means a missing key in an LLM response crashes at runtime. Go catches it at compile time (for known fields) and requires explicit error handling.

Where Python still wins

Python is irreplaceable for:

• Manim (QuantumSketch) — Python-only library with no Go equivalent
• ML model fine-tuning or training
• Data analysis and transformation (pandas, NumPy ecosystem)
• Libraries like sentence-transformers (reranking) — faster in Python than calling a Go wrapper

My rule: Python for where a specific library is irreplaceable. Go for everything else.

In QuantumSketch: Go handles the API, webhook receiving, job queuing, and Temporal workflow. Python handles Manim rendering and TTS integration only.

Go patterns for LLM backends

LLM client with retry and timeout

type LLMClient struct {
    httpClient *http.Client
    apiKey     string
    baseURL    string
}

func (c *LLMClient) Complete(ctx context.Context, req CompletionRequest) (string, error) {
    ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
    defer cancel()

    body, _ := json.Marshal(req)
    httpReq, _ := http.NewRequestWithContext(ctx, "POST", c.baseURL+"/completions", bytes.NewReader(body))
    httpReq.Header.Set("Authorization", "Bearer "+c.apiKey)
    httpReq.Header.Set("Content-Type", "application/json")

    var lastErr error
    for attempt := 0; attempt < 3; attempt++ {
        resp, err := c.httpClient.Do(httpReq)
        if err != nil {
            lastErr = err
            time.Sleep(time.Duration(attempt+1) * time.Second)
            continue
        }
        defer resp.Body.Close()

        if resp.StatusCode == 429 { // rate limit
            time.Sleep(time.Duration(attempt+1) * 2 * time.Second)
            continue
        }

        var result CompletionResponse
        json.NewDecoder(resp.Body).Decode(&result)
        return result.Content, nil
    }
    return "", fmt.Errorf("LLM failed after 3 attempts: %w", lastErr)
}

Fan-out: parallel LLM calls

func (s *Service) GenerateVariants(ctx context.Context, prompt string, n int) ([]string, error) {
    results := make([]string, n)
    errs := make([]error, n)
    var wg sync.WaitGroup

    for i := range n {
        wg.Add(1)
        go func(idx int) {
            defer wg.Done()
            results[idx], errs[idx] = s.llm.Complete(ctx, CompletionRequest{Prompt: prompt})
        }(i)
    }
    wg.Wait()

    for _, err := range errs {
        if err != nil { return nil, err }
    }
    return results, nil
}

Generating 5 variants in parallel: ~same latency as 1 sequential call.

FAQ

Why use Go instead of Python for AI backends? Go offers significantly better concurrency (goroutines vs Python threads/async), lower memory usage, static typing, and simpler deployment (single binary). For services that call LLM APIs rather than run models, Go outperforms Python substantially.

Does Go support all the AI libraries Python has? No. Python has irreplaceable ML libraries (PyTorch, Manim, transformers). Use Python where a specific library is required; use Go for API servers, orchestration, and business logic.

What's the goroutine vs Python async difference? Go goroutines are multiplexed over OS threads by the runtime — you write synchronous code and get concurrent execution automatically. Python async requires explicit async/await everywhere and still has the GIL for CPU-bound work. Go's model is simpler and more performant for I/O-heavy workloads like LLM API calls.

Can Go call Python services? Yes — in QuantumSketch, the Go orchestrator calls Python FastAPI services for Manim rendering and ML inference via HTTP. Polyglot microservices let each language do what it does best.

---

Written by Shihab Shahriar Antor — AI Engineer & Founder of Shahriar Labs. See also: Microservices as One Engineer · Rate Limiting APIs in Go.