Worker Pools with Channels

A worker pool is a common concurrency pattern where a fixed number of goroutines (workers) process jobs from a shared input channel. This bounds resource usage — you get concurrency without spawning an unbounded number of goroutines. The pool is created by launching N worker goroutines, each of which reads from a jobs channel and writes results to a results channel. The dispatcher sends jobs into the jobs channel and then closes it to signal that no more jobs are coming. Each worker loops over the jobs channel using a range loop, which exits automatically when the channel is closed. After all jobs have been dispatched, the dispatcher waits for all workers to finish (using a WaitGroup) and then closes the results channel. Worker pools are a building block for many real-world applications: processing uploaded files, making concurrent HTTP requests to external APIs, performing batch database operations, or any situation where you need to limit concurrency to avoid overwhelming downstream resources. Tuning the pool size is application-specific and often involves benchmarking.

Example

package main

import (
    "fmt"
    "sync"
    "time"
)

type Job struct {
    ID    int
    Input int
}

type Result struct {
    Job    Job
    Output int
}

func worker(id int, jobs <-chan Job, results chan<- Result, wg *sync.WaitGroup) {
    defer wg.Done()
    for job := range jobs {
        // Simulate CPU-bound work
        time.Sleep(10 * time.Millisecond)
        output := job.Input * job.Input
        fmt.Printf("worker %d processed job %d: %d^2 = %d\n",
            id, job.ID, job.Input, output)
        results <- Result{Job: job, Output: output}
    }
}

func main() {
    const (
        numWorkers = 3
        numJobs    = 10
    )

    jobs    := make(chan Job,    numJobs)
    results := make(chan Result, numJobs)

    var wg sync.WaitGroup
    for w := 1; w <= numWorkers; w++ {
        wg.Add(1)
        go worker(w, jobs, results, &wg)
    }

    // Dispatch jobs
    for j := 1; j <= numJobs; j++ {
        jobs <- Job{ID: j, Input: j}
    }
    close(jobs)

    // Close results after all workers finish
    go func() {
        wg.Wait()
        close(results)
    }()

    // Collect results
    total := 0
    for r := range results {
        total += r.Output
    }
    fmt.Println("sum of squares:", total)
}

HTML

Full Editor

Related Resources

Go Reference

Complete tag & property list

Go How-To Guides

Step-by-step practical guides

Go Exercises

Practice what you've learned