Consider a multi-agent system where agents perform a given task with different levels of ability. Agents are initially not aware of how well they perform in comparison with their peers, and are willing to self assess. This scenario is relevant, e.g., in wireless sensor networks, or in crowdsensing applications, where devices with embedded sensing capabilities collaboratively collect data to characterize the environment: the global performance is very sensitive to the measurement accuracy, and agents providing outliers should restrain to participate. This paper presents a distributed algorithm enabling each agent to self-assess its own ability. The algorithm tracks the outcomes of a local comparison test performed by pairs of agents when they randomly meet, and able to gauge their relative level of ability. The dynamics of the proportions of agents with similar assessments are described using continuous-time state equations. The existence of an equilibrium is shown. Closed-form expressions for the various proportions of agents with similar assessments are provided at equilibrium. In simulations, a community of agents equipped with sensors, and trying to determine the performance of their equipment is considered. Simulation results show a good fitting with theoretical predictions. (C) 2017 Elsevier Ltd. All rights reserved.