Researchers are looking for new ways to treat esophageal squamous cell carcinoma (ESCC). ESCC is a type of cancer that starts in certain cells that line the esophagus. The esophagus is the tube that connects the throat to the stomach. This study will look at ESCC that is either locally advanced unresectable, which means it has spread into tissue near where it started and cannot be completely removed by surgery, or metastatic, which means it has spread to other body parts. Available treatments for these types of ESCC include pembrolizumab and chemotherapy. Pembrolizumab is an immunotherapy, which is a treatment that helps the immune system fight cancer. Chemotherapy is medicine that destroys cancer cells or stops them from growing. Researchers want to learn about giving pembrolizumab and ifinatamab deruxtecan (I-DXd), a study medicine, with or without chemotherapy to treat ESCC. I-DXd is an antibody-drug conjugate (ADC). An ADC attaches to a protein on cancer cells and delivers treatment to destroy those cells. The main goal of this study is to learn about the safety of I-DXd and pembrolizumab with or without chemotherapy and if people tolerate them. Researchers also want to learn how cancer responds (gets smaller or goes away) to the study treatments.