We prove the following converse of the passivity theorem. Consider a causal system given by a sum of a linear time-invariant and a passive linear time-varying input-output map. Then, in order to guarantee stability (in the sense of finite L-2-gain) of the feedback interconnection of the system with an arbitrary nonlinear output strictly passive system, the given system must itself be output strictly passive. The proof is based on the S-procedure lossless theorem. We discuss the importance of this result for the control of systems interacting with an output strictly passive, but otherwise completely unknown, environment. Similarly, we prove the necessity of the small-gain condition for closed-loop stability of certain time varying systems, extending the well-known necessity result in linear robust control. (C) 2018 Elsevier Ltd. All rights reserved.