This is a phase 2 study whose purpose is see how useful investigational drug, avelumab, is in patients with locally advanced or metastatic penile cancer who are unfit for or progressed on platinum-based chemotherapy. The usefulness of avelumab in this study population will be determined by anti-tumor activity assessed by objective response rate. Avelumab is a monoclonal antibody that binds to a protein called programmed death-ligand 1 (PD-L1) and blocks its communication with another protein called programmed death protein 1 (PD-1). The communication of these proteins are thought to be important in the growth of tumors. Blocking these proteins from communicating may stop or shrink tumors.