In this article we study theoretically the capillary meniscus interaction between a floating submillimeter particle (attached to a fluid interface) and a vertical wall. It is shown that along with the particle weight and the upthrust, there is another force exerted on the particle which is caused by the deformation of the liquid meniscus. The latter we call "capillary image force" because of its similarity to the known image forces in electrostatics. In particular, the capillary image force can be both attractive and repulsive, depending on the specific boundary conditions at the three-phase contact line on the wall surface. Analytical expressions for the capillary interaction energy and force are derived by means of two alternative approaches and the agreement between them is very good. The theory predicts that under certain conditions the energy of interaction between the particle and the wall acquires a minimum; i.e., a stable equilibrium position of the particle at a finite separation from the wall should be observed. This prediction is experimentally proven in the second part of the study.