saral

Four ways to send a thousand notifications in Go

Most of us have written some version of this. You have a list of notifications, a Sender that talks to some provider, and a job: send them all, then tell me how many went out and which ones didn’t.

type Sender interface {
	Send(n Notification) error
}

type Report struct {
	Sent   int
	Failed []Failure // notification ID + the error
}

I’ve reached for goroutines on this kind of problem many times without being able to say why one shape was better than another. “Goroutines are cheap” is true, but it doesn’t tell you when to stop spawning them. So this post is me trying to answer that properly, starting from what the Go scheduler is actually doing with them.

What happens when you write go f()

Three letters do most of the work here: G, M and P.

  • G is a goroutine. It’s a small struct with a stack that starts at a few KB, not a thread.
  • M is an OS thread. This is the thing the kernel actually schedules.
  • P is a logical processor. It holds a queue of runnable Gs, and an M must hold a P to run Go code. The number of Ps is GOMAXPROCS, which defaults to your CPU count.

So on an 8 core machine you have 8 Ps, and at most 8 goroutines running Go code at any instant, no matter how many you’ve created. Each P has its own local run queue, and there’s a global one for overflow. When a P runs dry it steals half of another P’s queue, so work spreads itself out without you asking.

Global queue GGG M (OS thread) P0 G running local queue GGG M (OS thread) P1 G running local queue (empty) steals half Netpoller (parked on I/O) GGG Send() waits, G parks ready, requeued

Each M needs a P to run Go code. An idle P steals from a busy one, and a goroutine waiting on the network leaves the P entirely.

The part that matters for us is what happens when a goroutine blocks:

  • On network I/O (which is what Send almost certainly does), the goroutine is parked and the runtime’s netpoller watches the socket. The M immediately picks the next G from the P’s queue. No thread is wasted waiting.
  • On a blocking syscall or cgo call, the M is stuck along with its G. The runtime notices, detaches the P and hands it to another M (spinning up a new thread if needed), so the other goroutines keep going.

That’s why a single slow Send in a plain loop hurts so much. Nothing is wrong with the scheduler. You’ve simply given it one goroutine, so there’s nothing else to run while that one waits.

To compare the approaches fairly I wrote a fake sender where each Send sleeps 20 to 30ms, every seventh notification fails, and the first ten are “stuck” for 500ms (a bad stretch at the provider). The batch is 100 notifications. Timings below are from that run.

1. The plain loop

func DispatchSequential(s Sender, ns []Notification) Report {
	var r Report
	for _, n := range ns {
		if err := s.Send(n); err != nil {
			r.Failed = append(r.Failed, Failure{n.ID, err})
			continue
		}
		r.Sent++
	}
	return r
}

Took 7.3s. Total time is the sum of every Send, and the ten stuck ones alone cost 5 seconds.

  • Pro: trivial, no locks, easy to reason about, and a fine answer if the list is ten items.
  • Con: one goroutine means one G, so the scheduler has nothing to interleave. Every wait is dead time on a thread that could be doing other work.

2. A goroutine per notification

func DispatchSpawnAll(s Sender, ns []Notification) Report {
	var (
		wg sync.WaitGroup
		mu sync.Mutex
		r  Report
	)
	for _, n := range ns {
		wg.Add(1)
		go func() {
			defer wg.Done()
			err := s.Send(n)

			mu.Lock()
			defer mu.Unlock()
			if err != nil {
				r.Failed = append(r.Failed, Failure{n.ID, err})
				return
			}
			r.Sent++
		}()
	}
	wg.Wait()
	return r
}

Took 0.5s. Everything runs at once, so the whole batch takes as long as the slowest Send.

  • Pro: the shortest code that is also fast. Because Send parks on the netpoller, a few thousand goroutines sit idle almost for free.
  • Con: nothing bounds it. A million notifications means a million goroutines (a couple of GB of stacks), a million sockets competing for file descriptors, and a provider that rate limits you the moment the burst lands. The scheduler can handle it. Everything downstream of the scheduler often can’t.

If Send ever blocks in a syscall instead of parking, it gets worse: each stuck goroutine can pin a thread, and the runtime will keep creating Ms to keep the Ps busy.

3. Split the list into chunks, one worker each

func DispatchChunked(s Sender, ns []Notification, workers int) Report {
	var (
		wg sync.WaitGroup
		mu sync.Mutex
		r  Report
	)
	size := (len(ns) + workers - 1) / workers
	for start := 0; start < len(ns); start += size {
		chunk := ns[start:min(start+size, len(ns))]
		wg.Add(1)
		go func() {
			defer wg.Done()
			for _, n := range chunk {
				err := s.Send(n)

				mu.Lock()
				if err != nil {
					r.Failed = append(r.Failed, Failure{n.ID, err})
				} else {
					r.Sent++
				}
				mu.Unlock()
			}
		}()
	}
	wg.Wait()
	return r
}

Took 5.0s. Concurrency is capped at 10, which is good, but I rigged the first chunk to hold all ten stuck notifications, so one worker got stuck with 5 seconds of work while the other nine finished in about 250ms and went idle.

  • Pro: bounded concurrency, and no channel needed. Predictable if every Send costs about the same.
  • Con: the split is decided up front, before you know how long anything takes. The scheduler’s work stealing doesn’t rescue you here, and this is the bit that took me a while to see. Work stealing moves goroutines between Ps. It can’t move notifications between your chunks. The imbalance lives in your code, not in the runtime.

4. A shared channel and a pool of workers

func DispatchPool(s Sender, ns []Notification, workers int) Report {
	jobs := make(chan Notification)
	results := make(chan Failure)

	var wg sync.WaitGroup
	for range workers {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for n := range jobs {
				results <- Failure{n.ID, s.Send(n)}
			}
		}()
	}

	go func() {
		for _, n := range ns {
			jobs <- n
		}
		close(jobs)
		wg.Wait()
		close(results)
	}()

	var r Report
	for res := range results {
		if res.Err != nil {
			r.Failed = append(r.Failed, res)
			continue
		}
		r.Sent++
	}
	return r
}

Took 0.7s with the same 10 workers that took 5s in the chunked version. Workers pull the next notification the moment they’re free, so the ten stuck ones are spread across ten workers instead of piling onto one.

  • Pro: bounded concurrency and automatic load balancing. Also, only the collecting goroutine touches the report, so there’s no mutex. The channel is the synchronisation.
  • Con: more moving parts. You have to close the channels in the right order or you leak goroutines or deadlock, and you pick workers yourself. Too low and you’re back to being slow, too high and you’re approaching approach 2. For pure I/O work like this, the right number has more to do with what the provider tolerates than with your core count.

So which one

Approach Concurrency Handles uneven Send times Bounded Lines of ceremony
Loop 1 n/a yes none
Goroutine each N yes no little
Chunks fixed no yes some
Worker pool fixed yes yes most

If I had to pick one default, it’s the worker pool, and the reason is the same reason the scheduler works the way it does. Go’s runtime is good at taking a pool of runnable work and keeping every P busy. The pool pattern applies the same idea one level up: a shared queue, and workers that take from it when they’re free. The chunked version quietly breaks that by deciding the distribution before the work starts.

What I’d still want before putting this in production: a context.Context on Send so a stuck call can time out, and retries with backoff on the failures (since the report would otherwise list errors that were just a bad second).

I ran all four under go run -race and the numbers above are from a fake sender, so treat them as a shape, not a benchmark. Swap in your real provider and see if the shape holds. I’d be curious which approach your own code actually uses, and whether you chose it or just inherited it.

All posts