Parallel and Distributed Computing
Parallel and distributed computing harness multiple processors or machines to solve large problems faster. It is essential for training large models and scientific simulations.
Key Points
- Amdahl's law limits speedup due to sequential fractions.
- Data parallelism splits examples across workers; model parallelism splits parameters.
- Communication cost often dominates in distributed systems.
Formulas
Amdahl's law
$$S(n) = \frac{1}{(1-p) + \frac{p}{n}}$$
Speedup
$$S = \frac{T_1}{T_n}$$
Efficiency
$$E = \frac{S}{n}$$
Code Example
from multiprocessing import Pool
def square(x):
return x**2
with Pool(4) as p:
results = p.map(square, range(1000))
print(sum(results))