In this study, the aggregation dynamics of magic-sized CdTe quantum dots is investigated. The experiments show that the growth kinetics of the quantum dots is very sensitive to the Cd/Te ratio. The crossover from tellurium-rich to cadmium-rich conditions produces a very different aggregation pattern, which can be explained by the lack of formation of magic-sized nanoparticles during the reaction conditions. A simple simulation that includes both monomer-induced growth and aggregation growth is presented to reproduce the experimentally observed aggregation patterns. The simulation results strongly suggest that the experimental aggregation pattern can be reproduced if initially a double magic-sized distribution is assumed. The numerical data clearly show that the aggregation is enhanced by the dipole-dipole interaction. Simulations also suggest that the neck formation of the CdTe quantum dot aggregates is unlikely under the experimental conditions. The absence of neck formation is in agreement with the expectations during high-temperature synthesis.