• Patients must have histologically confirmed pancreatic cancer (excluding islets) (not otherwise specified \[NOS\]) (Medical Dictionary for Regulatory Activities \[MEDDRA\] code: 10033612).

• Patients must have unresectable locally advanced pancreatic cancer as determined by a multidisciplinary tumor board applying National Comprehensive Cancer Network (NCCN) version (v)2.2021 criteria or as surgically determined during failed resection attempt.

• Patients must have had prior first-line chemotherapy for this cancer for at least 16 weeks without clinical, biochemical, or radiologic progression. There should be a washout of at least 2 weeks from first-line chemotherapy and start of therapy on clinical trial.

• Age \>= 18 years. Because no dosing or adverse event data are currently available on the use of durvalumab and olaparib in combination with radiation in patients \< 18 years of age, children are excluded from this study.

• Body weight \> 30 kg.

• Eastern Cooperative Oncology Group (ECOG) performance status =\< 1 (Karnofsky \>= 70%).

• Hemoglobin \>= 9.0 g/dL without blood transfusion in last 4 weeks (within 2 weeks of enrollment).

• Absolute neutrophil count \>= 1,500/mcL (within 2 weeks of enrollment).

• Platelets \>= 100,000/mcL (within 2 weeks of enrollment).

• Total bilirubin =\< 1.5 x institutional upper limit of normal (ULN) (within 2 weeks of enrollment).

• Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\])/alanine transferase (ALT) (serum glutamic pyruvic transaminase) \[SGPT\]) =\< 2.5 x institutional ULN (within 2 weeks of enrollment).

• Creatinine =\< 1.5 x institutional ULN (within 2 weeks of enrollment).

• Measured creatinine clearance \> 60 mL/min/1.73 m\^2 (within 2 weeks of enrollment).

• Evidence of post-menopausal status or negative urinary or serum pregnancy test for female pre-menopausal patients. Women will be considered post-menopausal if they have been amenorrheic for 12 months without an alternative medical cause. The following age-specific requirements apply:

‣ Women \< 50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of exogenous hormonal treatments and if they have luteinizing hormone and follicle-stimulating hormone levels in the post-menopausal range for the institution or underwent surgical sterilization (bilateral oophorectomy or hysterectomy).

⁃ Women \>= 50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of all exogenous hormonal treatments, had radiation-induced menopause with last menses \> 1 year ago, had chemotherapy-induced menopause with last menses \> 1 year ago, or underwent surgical sterilization (bilateral oophorectomy, bilateral salpingectomy or hysterectomy).

• Life expectancy \>= 16 weeks.

• Patients who are human immunodeficiency virus (HIV) positive may participate IF they meet the following eligibility requirements:

‣ They must be stable on their anti-retroviral regimen with evidence of at least two undetectable viral loads within the past 6 months on the same regimen; the most recent undetectable viral load must be within the past 12 weeks.

⁃ They must have a CD4 count of greater than 250 cells/mcL over the past 6 months on this same anti-retroviral regimen and must not have had a CD4 count \< 200 cells/mcL over the past 2 years, unless it was deemed related to the cancer and/or chemotherapy induced bone marrow suppression.

∙ For patients who have received chemotherapy in the past 6 months, a CD4 count \< 250 cells/mcL during chemotherapy is permitted as long as viral loads were undetectable during this same chemotherapy.

⁃ They must have an undetectable viral load and a CD4 count \>= 250 cells/mcL within 7 days of enrollment.

⁃ They must not be currently receiving prophylactic therapy for an opportunistic infection and must not have had an opportunistic infection within the past 6 months.

⁃ HIV-infected patients should be monitored every 12 weeks for viral load and CD4 counts.

• For patients with evidence of chronic hepatitis B virus (HBV) infection, the HBV viral load must be undetectable on suppressive therapy, if indicated. Patients with a past or resolved HBV infection (defined as the presence of hepatitis B core antibody \[anti-HBc\] and absence of hepatitis B surface antigen \[HbsAg\]) are eligible.

• Patients with a history of hepatitis C virus (HCV) infection must have been treated and cured. For patients with HCV infection who are currently on treatment, they are eligible if they have an undetectable HCV viral load. Patients positive for HCV antibody are eligible only if polymerase chain reaction (PCR) is negative for HCV ribonucleic acid (RNA).

• Patients with a prior or concurrent malignancy whose natural history or treatment does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen are eligible for this trial.

• Willing to provide archived tissue, if available, from a previous biopsy. If tissue from initial biopsy is not available, a repeat biopsy is NOT required and patient will be eligible for enrollment.

• Patients must have radiographically measurable or evaluable disease (as per Response Evaluation Criteria in Solid Tumors \[RECIST\]v1.1).

• Must be able to tolerate CT and/or MRI with contrast.

• Patients with known history or current symptoms of cardiac disease, or history of treatment with cardiotoxic agents, should behave a clinical risk assessment of cardiac function using the New York Heart Association Functional Classification. To be eligible for this trial, patients should be class 2B or better.

• The effects of durvalumab and olaparib on the developing human fetus are unknown. For this reason and because radiotherapy used in this trial is known to be teratogenic, women of child-bearing potential and men must agree to use adequate contraception prior to study entry, for the duration of study participation, and for 6 months after the last dose of study treatment. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately. Male patients should not donate sperm throughout the period of taking olaparib and for 6 months following the last dose of olaparib. All females of childbearing potential (not surgically sterilized, and between menarche and 1 year post menopause) must have a negative screening pregnancy test.

• Ability to understand and the willingness to sign a written informed consent document. Participants with impaired decision-making capacity who have a legally-authorized representative (LAR) and/or family member available will also be eligible.