Inclusion Criteria:
* Histologically confirmed diagnosis of locally advanced or metastatic 1) naive small cell cancer of the bladder, urethra, or upper urinary tract, or 2) primary small cell or neuroendocrine prostate cancer will be enrolled in this study.
* Histological diagnosis of pure or mixed small cell or neuroendocrine cancer by a genitourinary pathologist is sufficient and confirmatory immunohistochemistry is not required.
* Cohort 1 will include subjects with no prior systemic chemotherapy for locally advanced or metastatic urothelial carcinoma, with the following exception(s):
* Platinum-based chemotherapy with recurrence \> 12 months from completion of therapy is permitted.
* Cohort 2 will include subjects with no prior systemic chemotherapy for primary small cell prostate cancer, with the following exception(s):
* Platinum-based chemotherapy with recurrence \> 12 months from completion of therapy is permitted.
* Cohort 2 will include subjects with prior treatments for metastatic castration-resistant prostate cancer (mCRPC) including:
* Prior chemotherapy with 2 other agents is allowed if \> 6 months elapsed from last dose (if docetaxel chemotherapy is used more than once for hormone-sensitive and for mCRPC it will be considered 1 therapy).
* Ongoing androgen deprivation therapy with up to 2 second-generation hormonal manipulations (e.g. including but not limited to abiraterone acetate and/or enzalutamide).
* Ongoing treatment with for bone metastasis (e.g. denosumab or zoledronic acid) is permitted.
* Prior immunotherapy with sipuleucel-T is allowed if completed \> 4 weeks prior to trial enrollment.
* A male participant must agree to use contraception during the treatment period and for at least 120 days after the last dose of pembrolizumab or 180 days after chemotherapy and refrain from donating sperm during this period.
* A female participant is eligible to participate if she is not pregnant, not breastfeeding, and at least one of the following conditions applies:
* Not a woman of childbearing potential (WOCBP) or
* A WOCBP who agrees to follow the contraceptive guidance during the treatment period and for at least 120 days after the last dose of pembrolizumab or 180 days after chemotherapy.
* The participant (or legally acceptable representative if applicable) provides written informed consent for the trial.
* Have measurable disease based on RECIST 1.1. Lesions situated in a previously irradiated area are considered measurable if progression has been demonstrated in such lesions.
* Have provided archival tumor tissue sample or newly obtained core or excisional biopsy of a tumor lesion not previously irradiated within 6 months of screening. Formalin-fixed, paraffin embedded (FFPE) tissue blocks are preferred to slides. In addition, the availability of fresh frozen tissue is encouraged. Newly obtained biopsies are preferred to archival tissue.
* Have an Eastern Cooperative Oncology Group (ECOG) performance status of 0 to 1. Evaluation of ECOG is to be performed within 7 days prior to the date of allocation.
* Absolute neutrophil count (ANC) \>= 1500/uL within 10 days prior to the start of study treatment.
* Platelets \>= 100 000/uL (microliter) within 10 days prior to the start of study treatment.
* Hemoglobin \>= 9.0 g/dL or \>= 5.6 mmol/L within 10 days prior to the start of study treatment.
* Criteria must be met without erythropoietin dependency and without packed red blood cell (pRBC) transfusion within last 2 weeks.
* Creatinine =\< 1.5 x upper limit of normal (ULN) OR measured or calculated creatinine clearance (glomerular filtration rate \[GFR\] can also be used in place of creatinine or creatinine clearance \[CrCl\]) \>= 30 mL/min for participant with creatinine levels \>1.5 x institutional ULN within 10 days prior to the start of study treatment.
* Creatinine clearance (CrCl) should be calculated per institutional standard.
* Total bilirubin =\< 1.5 ?ULN or direct bilirubin =\< ULN for participants with total bilirubin levels \> 1.5 ? ULN within 10 days prior to the start of study treatment.
* Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\]) and alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 ? ULN (=\< 5 ? ULN for participants with liver metastases) within 10 days prior to the start of study treatment.
* International normalized ratio (INR) or prothrombin time (PT) =\< 1.5 ? ULN unless participant is receiving anticoagulant therapy as long as PT or activated partial thromboplastin time (aPTT) is within therapeutic range of intended use of anticoagulants within 10 days prior to the start of study treatment.
Exclusion Criteria:
* Has disease suitable for local treatment with curative intent.
* A WOCBP who has a positive urine pregnancy test within 72 hours prior to receiving the first dose of trial medication. If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required.
* Has received prior therapy with an anti-PD-1, anti-PD-L1, or anti-PD-L2 agent or with an agent directed to another stimulatory or co-inhibitory T-cell receptor (e.g. CTLA-4, OX-40, CD137).
* Has received prior systemic anti-cancer therapy including investigational agents within 4 weeks prior to first dose of trial treatment.
* Note: Participants must have recovered from all AEs due to previous therapies to =\< grade 1 or baseline. Participants with =\< grade 2 neuropathy may be eligible.
* Note: If participant received major surgery, they must have recovered adequately from the toxicity and/or complications from the intervention prior to starting study treatment.
* Has received prior radiotherapy within 2 weeks of start of study treatment. Participants must have recovered from all radiation-related toxicities, not require corticosteroids, and not have had radiation pneumonitis. A 1-week washout is permitted for palliative radiation (=\< 2 weeks of radiotherapy) to non-central nervous system (CNS) disease.
* Has received a live vaccine within 30 days prior to the first dose of study drug. Examples of live vaccines include, but are not limited to, the following: measles, mumps, rubella, varicella/zoster (chicken pox), yellow fever, rabies, Bacillus Calmette?Guerin (BCG), and typhoid vaccine. Seasonal influenza vaccines for injection are generally killed virus vaccines and are allowed; however, intranasal influenza vaccines (eg, FluMist) are live attenuated vaccines and are not allowed.
* Is currently participating in or has participated in a study of an investigational agent or has used an investigational device within 4 weeks prior to the first dose of study treatment.
* Note: Participants who have entered the follow-up phase of an investigational study may participate as long as it has been 4 weeks after the last dose of the previous investigational agent.
* Has a diagnosis of immunodeficiency or is receiving chronic systemic steroid therapy (in dosing exceeding 10 mg daily of prednisone equivalent) or any other form of immunosuppressive therapy within 7 days prior to the first dose of study drug.
* Has a known additional malignancy that is progressing or has required active treatment within the past 3 years. Note: Participants with basal cell carcinoma of the skin, squamous cell carcinoma of the skin, or carcinoma in situ (e.g. breast carcinoma, cervical cancer in situ) that have undergone potentially curative therapy are not excluded.
* For Cohort 1, a history of prostate cancer that was identified incidentally following cystoprostatectomy for bladder cancer is acceptable provided that the PSA is \< 0.2.
* Has known active CNS metastases and/or carcinomatous meningitis. Participants with previously treated brain metastases may participate provided they are radiologically stable, i.e. without evidence of progression for at least 4 weeks by repeat imaging (note that the repeat imaging should be performed during study screening), clinically stable and without requirement of steroid treatment for at least 14 days prior to first dose of study treatment.
* Has severe hypersensitivity (\>= grade 3) to pembrolizumab and/or any of its excipients.
* Has active autoimmune disease that has required systemic treatment in the past 2 years (i.e. with use of disease modifying agents, corticosteroids or immunosuppressive drugs). Replacement therapy (eg., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment.
* Has a history of (non-infectious) pneumonitis that required steroids or has current pneumonitis.
* Has an active infection requiring systemic therapy.
* Has a known history of human immunodeficiency virus (HIV).
* Has a known history of hepatitis B (defined as hepatitis B surface antigen \[HBsAg\] reactive) or known active hepatitis C virus (defined as HCV ribonucleic acid \[RNA\] \[qualitative\] is detected) infection. Note: no testing for hepatitis B and hepatitis C is required unless mandated by local health authority.
* Has a known history of active TB (bacillus tuberculosis).
* Has a history or current evidence of any condition, therapy, or laboratory abnormality that might confound the results of the study, interfere with the subject?s participation for the full duration of the study, or is not in the best interest of the subject to participate, in the opinion of the treating investigator.
* Has known psychiatric or substance abuse disorders that would interfere with cooperation with the requirements of the trial.
* Is pregnant or breastfeeding, or expecting to conceive or father children within the projected duration of the study, starting with the screening visit through 120 days after the last dose of trial treatment.
