Five Go Concurrency Patterns, every Go Developer must know - Worker Pool
Parallel processing of tasks with efficient system resource utilization
In this three part series, we will be mastering `Go Concurrency` by exploring top Go Concurrency patterns that are frequently used in solving well known concurrency problems efficiently.
Series:
Worker Pool
In the current part, we will explore in depth on `Worker Pool Pattern` by applying it in the real world industry examples.
The Worker Pool pattern effectively leverages goroutines and channels to distribute tasks or jobs to a pool of worker goroutines. The main goals are to limit the number of concurrent goroutines and efficiently manage task execution, avoiding the overhead or risk of running an excessive number of goroutines concurrently.
It is particularly useful in situations where tasks need to be processed concurrently but system resources are limited, such as CPU, memory, or network bandwidth.
We will apply this pattern to handle a `Banking transactions processing` scenario.
Scenario
In the banking industry, consider the need to process thousands of financial transactions. Each transaction involves say:
Reading data from a database.
Performing validations (e.g., fraud checks, balance validation).
Updating account balances.
Creating a new goroutine for each transaction may lead to resource exhaustion (e.g., too many open connections or high memory usage). so, we will apply the Worker Pool pattern to process transactions concurrently but with a controlled number of workers.
Solution:
We setup three main constructs to solve the problem.
Define a minimilistic `Transaction` struct
Define a `ProcessTransaction function` that simulates processing a transaction and return if there is any error
Define a `worker function` that processes the list of tasks retrieved from a channel and sends out the result
Define a `main function` to orchestrate the whole process
ProcessTransaction : receives a transaction instance and simulate processing it by introducing a random delay. At a random time, it also returns a error simulation. Finally prints as Transaction completed.
Worker: this constitues the worker goroutine in the pool. It ranges transaction from the tasks channel (behaves as task queue) and sends it for processing by calling `ProcessTransaction`. Writes error to the results channel when a error is returned from `ProcessTransaction`. WaitGroup is important to make the wait function to wait till all workers completes its processing.
Main funtion performs following steps:
Defines a set of sample Transactions.
Define a variable `numWorkers` and initializes to 3 - which means we want to restrict spinning up only 3 worker goroutines in the pool.
We start 3 worker goroutines and pass transactions via channel to those workers
We range transactions slice and write transaction one by one on the tasks channel. Meanwhile each worker started in the above step is blocked and waiting to receive a task from the task channel (as the channel is a unbuffered channel). When a task is available in the channel any random worker picks up the task and goes for processing. Once all the transactions are written on channel, then we close the tasks channel.
We wait on waitgroup again on a separate goroutine, when waitgroup over, then we close result channel
We read error result from the result channel and displays when any error occured.
There are multiple goroutine dynamics in this program. Main function starts the process and spinup all the separate goroutines and run till it gets blocked on result channel in the last block of code. Everything else runs on random unpredictable order, in parallel. When all the worker completes its work, then waitgroup completes and then closes the result channel. Closing of result channel is the trigger to complete reading result from the result channel, then terminates for range.
Sample Output:
The above output result is one sample run, but the order of execution can drastically varry from one run to another due to parallel nature of running goroutines (when multiple cpu core available). But the main program always waits to complete all the worker goroutines.
