Inclusion Criteria:
* Presence of histologically confirmed, advanced or metastatic Merkel cell carcinoma (MCC), which is considered incurable with standardly available therapies
* Presence of at least one MCC tumor, considered measurable per Response Evaluation Criteria in Solid Tumors (RECIST) version (v)1.1
* Age 18 or older. (NOTE: Both men and women, and members of all races and ethnic groups are eligible for this trial.)
* Eastern Cooperative Oncology Group (ECOG) performance score of 0-2
* Must have previously received at least one prior systemic treatment regimen with an anti-PD-(L)1 agent (administered as monotherapy or in combination with another treatment)
* Must meet the following criteria defining anti-PD-(L)1 refractory MCC: Best response of progressive disease (PD) or development of PD after best response of complete response (CR), partial response (PR), or stable disease (SD) after receiving at least 6 weeks of therapy with an anti-PD-(L)1 agent; PD must develop within 6 months of the last administration of anti-PD-(L)1 agent
* Absolute neutrophil count (ANC) \>= 1 x 10\^9/L
* Platelet count \>= 100 × 10\^9/L
* Hemoglobin \>= 9 g/dL (may have been transfused)
* Alanine aminotransferase (ALT), aspartate aminotransferase (AST) and total bilirubin level =\< 2 x the upper limit of normal (ULN) (or total bilirubin =\< 2.5 x ULN in patients with Gilbert's syndrome, and AST, ALT =\< 2.5 x ULN in patients with hepatic metastases)
* Estimated creatinine clearance \>= 30mL/min according to the Cockcroft-Gault formula or according to local institutional standard
* Activated partial thromboplastin time (aPTT) and international normalized ratio (INR) or prothrombin time (PT) =\< 1.5 x ULN unless on therapeutic anticoagulants
* Troponin I (TnI) =\< institutional ULN. (Note: Patients with unexplained elevated TnI at baseline may undergo a cardiac evaluation and be considered for treatment following a discussion with the principal investigator or designee.)
* Must consent to undergo serial tumor biopsies at study defined timepoints, unless deemed unsafe or technically not feasible by the study investigator
* Must have an ability to understand and the willingness to sign a written informed consent document
* Women of childbearing potential must have a negative serum or urine pregnancy test at screening
* Both male and female subjects must be willing to use highly effective contraception, as stipulated in national or local guidelines, throughout the study and for at least 180 days after last treatment administration, if the risk of conception exists
Exclusion Criteria:
* Residual adverse event(s) from prior therapy grade \> 1 (National Cancer Institute \[NCI\]-Common Terminology Criteria for Adverse Events \[CTCAE\] v 5.0) that could interfere with study endpoints or put patient safety at risk
* Known active central nervous system (CNS) metastases and/or prior history of carcinomatous meningitis. (NOTE: Participants with previously treated brain metastases may participate provided that they are stable, without evidence of progression by brain imaging performed within the screening period and at least 4 weeks after the treatment of brain metastases, and any neurologic symptoms must have stabilized. Patients must not have any evidence of new or enlarging brain metastases or increasing CNS edema and must not have required steroids for this purpose for at least 7 days before the first dose of study treatment.)
* History of serious immune-related adverse events (IRAEs) from prior immunotherapy that resulted in permanent discontinuation of anti-PD-(L)1 and could jeopardize patient safety with the investigational regimen. (NOTE: Any prior grade 2 or higher IRAE must be discussed with the Principal Investigator or designee for risk/benefit assessment.)
* Known allergy or hypersensitivity to any component of the study drugs formulation (including excipients and additives) that could interfere with study endpoints or put patient safety at risk
* Previous malignant disease (other than MCC) diagnosed within 3 years from day 1 of study treatment that could interfere with study endpoints or put patient safety at risk. (NOTE: Exception will be made for adequately treated basal or squamous cell carcinoma of the skin or carcinoma in situ (skin, bladder, cervical, colorectal, breast) or low grade prostatic intraepithelial neoplasia or grade 1 prostate cancer. Any other neoplasm, which has been treated adequately and is adjudged by the treating investigator to have a low risk of recurrence during the study, could be enrolled only after written approval from the principal investigator or designee.)
* Known active hepatitis B virus (HBV) or hepatitis C virus (HCV), defined as follows:
* Active HBV is defined as a known positive hepatitis B virus surface antigen (HBsAg) result or positive total hepatitis B virus core antibody (anti-HBc) results in the absence of hepatitis B virus surface antibody (anti-HBsAb). NOTE: When HBsAg is negative and hepatitis B virus core antibody (HBcAb) is positive, HBV-deoxyribonucleic acid (DNA) should be measured. When HBV-DNA is negative, this participant could be enrolled with close monitoring of HBV activities.)
* Active HCV is defined as a known positive HCV antibody result and quantitative HCV-ribonucleic acid (RNA) results greater than the lower limits of detection of the assay. (NOTE: Participants who have had definitive treatment for HCV are permitted if HCV-RNA is undetectable.)
* Known uncontrolled human immunodeficiency (HIV) infection. (NOTE: HIV-positive patients may be allowed if all of the following criteria are met: CD4 count \>= 300/uL, undetectable viral load, receiving antiretroviral therapy, and risk/benefit ratio is deemed favorable when considering enrollment.)
* Known active autoimmune disease, allograft requiring systemic immunosuppression, or other condition requiring chronic systemic corticosteroids (\> 10 mg/day of prednisone or equivalent). (NOTE: Exceptions will be made for patients with autoimmune conditions such as diabetes type I, vitiligo, psoriasis, or hypo- or hyperthyroid diseases not requiring immunosuppressive treatment; patients receiving physiologic corticosteroid replacement therapy at doses =\< 10 mg/day of prednisone or equivalent for adrenal or pituitary insufficiency; patients with a condition such as asthma or chronic obstructive pulmonary disease that requires intermittent use of bronchodilators, inhaled steroids, or local steroid injections; or those who required brief courses of corticosteroids for prophylaxis (e.g., contrast dye allergy) or study treatment-related standard premedication. Any other situation must be discussed with the principal investigator or designee for risk/benefit assessment.)
* Immunosuppressed status due to severe uncontrolled diabetes, concurrent hematological malignancy, or other comorbidities
* Known history of serious, active infections (aside from well-controlled HIV) requiring systemic antimicrobial agents within 14 days before the first dose of study treatment. (NOTE: Chronic infections such as herpes simplex virus requiring suppressive therapy may be allowed after discussion with the Principal Investigator or designee for risk/benefit assessment)
* Known history of clinically significant interstitial lung disease, or active noninfectious pneumonitis
* Clinically significant (i.e., active) cardiovascular disease such as cerebral vascular accident or myocardial infarction (within 6 months prior to first dose of study treatment), ongoing unstable angina or congestive heart failure ( \>= New York Heart Association Classification class II), or serious cardiac arrhythmia that could jeopardize patient safety on the study
* Receipt of live vaccine(s) within 30 days of planned start of study treatment. (NOTE: Examples of live vaccines include but are not limited to measles, mumps, rubella, varicella-zoster \[chickenpox), yellow fever, rabies, bacillus calmette-guerin \[BCG\], and typhoid vaccines. Seasonal influenza vaccines for injection are generally killed-virus vaccines and are allowed; however, intranasal influenza vaccines are live, attenuated vaccines and are not allowed)
* Known severe acute or chronic medical conditions such as uncontrolled seizure disorder, serious psychiatric illness, or laboratory abnormalities, that may increase the risk associated with study participation or may interfere with the interpretation of study endpoints and, in the judgment of the investigator, would make the patient inappropriate for entry into this study
* Pregnant or breast-feeding woman
