This work reinforced both a glassy polymer (high stiffness matrix) and a rubbery polymer (low stiffness matrix) with cellulose nanocrystals (CNC) derived from natural sources. CNC addition always increased stiffness while it increased toughness for a rubbery polymer and caused no loss in toughness for a glassy polymer. These results contradict many claims that when stiffness increases, the toughness decreases. We show that these claims depend on how toughness is measured. Our results were based on toughness measured using the essential work of fracture method. In contrast, toughness determined from area under the stress-strain curve shows a significant decrease, but that method may be a poor measure of toughness. Property enhancements usually require a good fiber/matrix interface. We used modeling of stiffness properties to confirm that CNC has a good interface with the studied polymer matrices.