Ray effect is one of the major shortcomings of discrete ordinate method (DOM). It is a consequence of angular discretization, resulting in unphysical oscillations in the solutions, and often occurs in the problems where the temperature of walls is discontinuous. In this paper, the modified DOM (MDOM) is developed for the radiative heat transfer in an axisymmetric cylindrical enclosure, which contains emitting, absorbing and scattering medium, to overcome this shortcoming. The radiative intensity is decomposed into two components, the wall-related and the medium-related ones. The former is determined analytically, whereas the latter is still solved by DOM. In MDOM, the most important step is the evaluation of wall-related incident radiation and radiative heat flux. Two methods to evaluate them are comparatively given: one is the semi-analytical method based on coordinate transformation which is more accurate, and the other is the numerical method with segments integration to treat the discontinuity which is mathematically simpler. Results show that both are effective to mitigate the ray effect, and cost computational time comparable to DOM. Exact semi-analytic solutions for the pure absorbing cases are also given in tabular form for convenience. Besides, alternative boundary conditions for the MDOM proposed by Sakami and Charette (JQSRT, 64(2000) 275-298) are investigated. This type of boundary conditions is proved to have simplified solution procedure, but results in slight unphysical oscillations in the MDOM solutions. (C) 2019 Elsevier Ltd. All rights reserved.