• Patients must have a histologically confirmed diagnosis non-prostate GU cancer

• Patients with the presence of cancer-associated genetic mutations in one or more pathogenic or likely pathogenic gene alterations tested in the FoundationOne FoundationOne®CDx (F1CDx) panel will be enrolled in cohorts 1 or 2 as follows:

‣ Cohort 1: BRCA1, BRCA2, ATM, BAP1, PALB2, and BRIP1, or tumor mutational burden (TMB) where 10 or greater mutations/megabase

⁃ ABL1, FANCE, POLD1, ATR, FANCG, POLE, ATRX, FANCL, RAD51, BARD1, IKBKE, SMARCB1, BRD4, MEN1, STK11, CCND1, MLH1, TP53, CHEK1, MSH2, CHEK2, MSH6, DOT1L, MUTYH, FANCA, NPM1, FANCC, PMS2

• Patients with benign or variants of unknown significance as determined by FoundationOne FoundationOne®CDx (F1CDx) panel and Genetics Review Panel review will be enrolled in Cohort 3 to be followed for survival

• Foundation One mutation analysis results performed prior to enrollment on this study may be accepted for eligibility review and in the event that a patient cannot undergo a biopsy and tumor is not available, Foundation Medicine liquid biopsy may be performed

• Patients must have measurable disease, defined as at least one lesion that can be accurately measured in at least one dimension (longest diameter to be recorded for non-nodal lesions and short axis for nodal lesions) as \>= 20 mm (\>= 2 cm) by chest x-ray or as \>= 10 mm (\>= 1 cm) with CT scan, MRI, or calipers by clinical exam

• Evidence of disease progression as defined by Response Evaluation Criteria in Solid Tumors (RECIST) (version 1.1) during treatment or after the most recent dose of therapy with at least one platinum-based regimen of chemotherapy and/or an immune-checkpoint inhibitor (atezolizumab, pembrolizumab, nivolumab, avelumab or durvalumab) (2-week washout from chemotherapy and 4-weeks washout from monoclonal antibodies is required)

• Age \>= 18 years. Because no dosing or adverse event data are currently available on the use of olaparib in patients \< 18 years of age, children are excluded from this study, but will be eligible for future pediatric trials

• Eastern Cooperative Oncology Group (ECOG) performance status =\< 1 (or Karnofsky \>= 70%)

• Leukocytes \>= 3,000/mcL

• Absolute neutrophil count \>= 1,500/mcL

• Platelets \>= 100,000/mcL

• Total bilirubin =\< 1.5 x upper limit of normal (ULN) (for subjects with documented Gilbert's disease total bilirubin =\< 3.0 mg/dL)

• Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x institutional upper limit of normal (for subjects with liver metastasis AST/ALT =\< 5 x ULN)

• Creatinine clearance \>= 50 mL/min/1.73 m\^2

• Hemoglobin \>= 9 g/dL; transfusions are allowed

• Prothrombin time (PT)/international normalized ratio (INR) and activated partial thromboplastin time (aPTT) within 1.25 x ULN institutional limits, except where a lupus anti-coagulant has been confirmed or the patient is on direct oral anticoagulant (DOA)

• Patients must be able to tolerate oral medications and not have gastrointestinal illnesses that would preclude absorption of olaparib

• Pre-clinical data indicate that olaparib adversely affects embryofetal survival and development. Therefore, women of child-bearing potential and their partners should agree to use two (2) highly effective forms of contraception throughout study participation and for at least six (6) months after the last dose of olaparib. Male study participants should avoid fathering a child or donating sperm during the study and for three (3) months after the last dose of olaparib.

‣ Note: Olaparib is a PARP inhibitor with the potential for teratogenic or abortifacient effects. Because there is a potential risk for adverse events in nursing infants secondary to treatment of the mother with olaparib, breastfeeding should be discontinued if the mother is treated with olaparib

• Ability to understand and the willingness to sign a written informed consent document or patients with impaired decision making capacity (IDMC) if they are represented by a legally authorized representative (LAR)

• Patients must provide archival tumor sample for mutation analysis or be willing to undergo mandatory screening biopsy. In the event that the patient cannot undergo biopsy and tumor is not available, Foundation Medicine liquid biopsy will be performed

• Postmenopausal or evidence of non-childbearing status for women of childbearing potential: negative urine or serum pregnancy test within 28 days of study treatment at screening. Note: This will also need to be confirmed within 3 days prior to treatment on day 1. Postmenopausal is defined as at least one (1) of the following:

‣ Amenorrheic for 1 year or more following cessation of exogenous hormonal treatments

⁃ Luteinizing hormone (LH) and follicle stimulating hormone (FSH) levels in the post menopausal range for women under 50

⁃ Radiation-induced oophorectomy with last menses \> 1 year ago

⁃ Chemotherapy-induced menopause with \> 1 year interval since last menses

⁃ Surgical sterilization (bilateral oophorectomy or hysterectomy)

• Patients is willing and able to comply with the protocol for the duration of the study including undergoing treatment and scheduled visits and examinations