Recurrent, metastatic prostate cancer continues to be a leading cause of cancer-death in men. The androgen receptor (AR) is a modular, ligand-inducible transcription factor that regulates the expression of genes that can drive the progression of this disease, and as a consequence, this receptor is a key therapeutic target for controlling prostate cancer. The current drugs designed to directly inhibit the AR are called anti-androgens, and all act by competing with androgens for binding to the androgen/ligand binding site. Unfortunately, with the inevitable progression of the cancer to castration resistance, many of these drugs become ineffective. However, there are numerous other regulatory sites on this protein that have not been exploited therapeutically. The regulation of AR activity involves a cascade of complex interactions with numerous chaperones, co-factors and co-regulatory proteins, leading ultimately to direct binding of AR dimers to specific DNA androgen response elements within the promoter and enhancers of androgen-regulated genes. As part of the family of nuclear receptors, the AR is organized into modular structural and functional domains with specialized roles in facilitating their inter-molecular interactions. These regions of the AR present attractive, yet largely unexploited, drug target sites for reducing or eliminating androgen signaling in prostate cancers. The design of small molecule inhibitors targeting these specific AR domains is only now being realized and is the culmination of decades of work, including crystallographic and biochemistry approaches to map the shape and accessibility of the AR surfaces and cavities. Here, we review the structure of the AR protein and describe recent advancements in inhibiting its activity with small molecules specifically designed to target areas distinct from the receptor's androgen binding site. It is anticipated that these new classes of anti-AR drugs will provide an additional arsenal to treat castration-resistant prostate cancer.