Inclusion Criteria:
* Newly diagnosed pathologically-confirmed locally advanced rectal cancer (defined by 8th edition AJCC stage 2 or 3, or stage 1 not eligible for sphincter-sparing surgery) with plans to proceed with total neoadjuvant short course radiation as part of their neoadjuvant therapy as confirmed by treating physician
* At least 18 years of age.
* ECOG performance status 0, 1, or 2
* Adequate bone marrow and organ function as defined below:
* Absolute neutrophil count ≥ 1.5 K/cumm
* Platelets ≥ 100 K/cumm
* Hemoglobin \> 9 g/dL
* Total bilirubin ≤ 1.5 x IULN
* AST(SGOT)/ALT(SGPT) ≤ 2.5 x IULN
* Serum creatinine \< 1.5 x IULN OR measured or calculated creatinine clearance ≥ 50 mL/min/1.73 m2
* Applicable to subjects enrolled at Washington University and Dana Farber Cancer Institute only: Willing to undergo study-related biopsies subject to accessibility of tumor, appropriateness of biopsy (not contraindicated), and continued subject consent. If biopsy is not safe and feasible per treating physician, then patient may still enroll with permission of sponsor-investigator.
* Women of childbearing potential and men must agree to contraceptive methods as described in protocol prior to study entry, for the duration of study participation, and for 120 days after the last dose of study treatment. Should a woman become pregnant or suspect she is pregnant while participating in this study, she must inform her treating physician immediately.
* Able to understand and willing to sign an IRB approved written informed consent document (or that of legally authorized representative, if applicable).
Exclusion Criteria:
* Received prior anti-cancer therapy for rectal cancer.
* Prior treatment with agents targeting IDO pathway (including indoximod)
* Previous radiotherapy in the pelvic region or previous rectal surgery (e.g. TEM) or any investigational treatment for rectal cancer within the past month.
* Known or suspected presence of another malignancy that could be mistaken for the malignancy under study during disease assessments.
* Currently receiving any other investigational agents.
* Extensive growth into cranial part of the sacrum (above S3) or the lumbosacral nerve roots indicating that surgery will never be possible even if substantial tumor downsizing is seen.
* Presence of metastatic disease or recurrent rectal tumor.
* Diagnosis of Familial Adenomatosis Polyposis coli (FAP), Hereditary Non-Polyposis Colorectal Cancer (HNPCC), active Crohn's disease, or active ulcerative colitis.
* A history of allergic reactions attributed to compounds of similar chemical or biologic composition to epacadostat, pembrolizumab, 5-FU, oxaliplatin, or other agents used in the study.
* Has an active infection requiring systemic therapy.
* Warfarin (Coumadin): patients currently on warfarin are excluded. Patients who go off warfarin and have INR within normal limits have no washout period.
* Any history of serotonin syndrome (SS) after receiving serotonergic drugs. This syndrome has been most closely associated with the use of MAOIs, meriperidine, linezolid, or methylene blue; all of these agents are prohibited during the study
* Uncontrolled intercurrent illness including, but not limited to active tuberculosis infection, pneumonitis requiring treatment, symptomatic congestive heart failure, unstable angina pectoris, or cardiac arrhythmia.
* Has an active or inactive autoimmune disease or syndrome (i.e. rheumatoid arthritis, moderate or severe psoriasis, multiple sclerosis, inflammatory bowel disease) that has required systemic treatment in the past 2 years or is receiving systemic therapy for an autoimmune or inflammatory disease (i.e. with use of modifying agents, corticosteroids, or immunosuppressive drugs). Exceptions include subjects with vitiligo or resolved childhood asthma/atopy, hypothyroidism stable on hormone replacement, controlled asthma, Type I diabetes, Graves' disease, or Hashimoto's disease.
* An abnormal screening ECG that, in the investigator's opinion, is clinically meaningful.
* Presence of a gastrointestinal condition that may affect drug absorption.
* Receipt of live attenuated vaccine within 30 days before the first dose of study treatment. Examples of live vaccines include, but are not limited to, the following: measles, mumps, rubella, chicken pox, yellow fever, rabies, Bacillus Calmette-Guérin (BCG), and typhoid vaccine. Seasonal influenza vaccines for injection are generally killed virus vaccines and are allowed; however, intranasal influenza vaccines (e.g. FluMist) are live attenuated vaccines and are not allowed.
* Pregnant and/or breastfeeding. Women of childbearing potential must have a negative pregnancy test within 7 days of planned start of study therapy.
* Known active hepatitis B (e.g. HBsAg reactive or HBV DNA detected) or hepatitis C (e.g. HCV RNA \[qualitative\] is detected) infection. Testing at screening is required (Serology testing with HBsAg, HBsAb, and HCV Ab are required; HBV DNA or HCV RNA are only required in the setting of serology tests compatible with possible active infection.).
* Known microsatellite instability- high (MSI-H) or mismatch repair deficient rectal cancer.
