The ability to detect light over a broad spectral range is central to practical optoelectronic applications and has been successfully demonstrated with photodetectors of two-dimensional layered crystals such as graphene and MoS2. However, polarization sensitivity within such a photodetector remains elusive. Here, we demonstrate a broadband photodetector using a layered black phosphorus transistor that is polarization-sensitive over a bandwidth from similar to 400 nm to 3,750 nm. The polarization sensitivity is due to the strong intrinsic linear dichroism, which arises from the in-plane optical anisotropy of this material. In this transistor geometry, a perpendicular built-in electric field induced by gating can spatially separate the photogenerated electrons and holes in the channel, effectively reducing their recombination rate and thus enhancing the performance for linear dichroism photodetection. The use of anisotropic layered black phosphorus in polarization-sensitive photodetection might provide new functionalities in novel optical and optoelectronic device applications.