Singlet exciton fission allows the fast and efficient generation of two spin triplet states from one photoexcited singlet. It has the potential to improve organic photovoltaics, enabling efficient coupling to the blue to ultraviolet region of the solar spectrum to capture the energy generally lost as waste heat. However, many questions remain about the underlying fission mechanism. The relation between intermolecular geometry and singlet fission rate and yield is poorly understood and remains one of the most significant barriers to the design of new singlet fission sensitizers. Here we explore the structure-property relationship and examine the mechanism of singlet fission in aggregates of astaxanthin, a small polyene. We isolate five distinct supramolecular structures of astaxanthin generated through self-assembly in solution. Each is capable of undergoing intermolecular singlet fission, with rates of triplet generation and annihilation that can be correlated with intermolecular coupling strength. In contrast with the conventional model of singlet fission in linear molecules, we demonstrate that no intermediate states are involved in the triplet formation: instead, singlet fission occurs directly from the initial 1Bu photoexcited state on ultrafast time scales. This result demands a re-evaluation of current theories of polyene photophysics and highlights the robustness of carotenoid singlet fission.