A composite contact algorithm for a complex particle system containing polyhedrons and super-ellipsoids is proposed in this paper to simulate the grinding process of ore in a ball mill. In this algorithm, the contacts between polyhedrons are calculated based on the overlapping volume method, and the contacts between super-ellipsoids are determined by a deepest point method. The contacts between super-ellipsoids and polyhedrons are obtained by transforming super-ellipsoids into polyhedrons, and then using the overlapping volume method between the polyhedrons. The increase in the number of faces of the super-ellipsoid contributes to the closer shape to the real shape. In this paper, experimental researches and numerical simulations are carried out to validate the accuracy of this composite contact algorithm. Furthermore, the influence of the ratio of super-ellipsoids and polyhedrons, the number of faces of converted super-ellipsoid, and the sharpness of the super-ellipsoid on the computational efficiency are compared in both box filling process and particles mixing process in a rotating tumbler. Finally, the stability and feasibility of this composite contact algorithm is tested in a ball mill application containing more particles. (C) 2020 Elsevier B.V. All rights reserved.