Inclusion Criteria:
* Documented informed consent of the participant and/or legally authorized representative.
* Age: \>= 18 years.
* Eastern Cooperative Oncology Group (ECOG) =\< 2.
* Life expectancy \> 3 months.
* Patients with histologically confirmed acute myeloid leukemia (AML), according to World Health Organization (WHO) criteria, with relapsed/refractory disease.
* Patients must have any one of the following treatment history criteria:
* Relapsed AML
* Failed at least 1 line of salvage therapy or
* Untreated relapse and are not candidates for allogeneic hematopoietic stem cell transplantation (alloHCT)
* De novo AML
* have not achieved complete response (CR) after 2 lines of therapy or
* refractory to frontline therapy and not eligible for alloHCT
* AML evolving from myelodysplastic syndrome (MDS) or myeloproliferative disorder who have failed hypomethylating agents (HMA) or induction chemotherapy
* Patients who have relapsed after allo-HCT are eligible if they are at least 3 months after HCT, do not have active graft versus host disease (GVHD) and are off immunosuppression except for maintenance dose of steroids (prednisone 10 mg/day or less).
* Male subjects must agree to not donate sperm while taking protocol therapy through at least 90 days after the last dose.
* White blood cell (WBC) =\< 25 x 10\^9/L prior to initiation of venetoclax. Cytoreduction with hydroxyurea prior to treatment and/or during cycle 1 may be required.
* Total bilirubin =\< 1.5 X upper limit of normal (ULN) (unless has Gilbert's disease).
* Aspartate aminotransferase (AST) =\< 2.5 x ULN.
* Alanine aminotransferase (ALT) =\< 2.5 x ULN.
* Creatinine clearance of \>= 50 mL/min per 24 hour urine test or the Cockcroft-Gault formula.
* QTc =\< 480 ms.
* Women of childbearing potential (WOCBP): negative urine or serum pregnancy test. If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required.
* Agreement by females and males of childbearing potential\* to use an effective method of birth control or abstain from heterosexual activity for the course of the study through at least 6 months (females) and 3 months (males) after the last dose of protocol therapy.
* Childbearing potential defined as not being surgically sterilized (men and women) or have not been free from menses for \> 1 year (women only).
Exclusion Criteria:
* Current or planned use of other investigational agents, antineoplastic, biological, chemotherapy, or radiation therapy during the study treatment period, or within 2 weeks prior to day 1 of protocol therapy, with the following exception:
* Hydroxyurea which may be continued through cycle 1.
* Expected to undergo HCT within 120 days of enrollment.
* Current or planned use of agents that prolong or suspected to prolong QTc.
* Received strong or moderate CYP3A inducers or St. John's Wort within 7 days prior to day 1 of protocol therapy.
* Received strong or moderate CYP3A inhibitors, or consumed grapefruit, grapefruit products, Seville oranges (including marmalade containing Seville oranges) or Star fruit within 3 days prior to day 1 of protocol therapy.
* P-glycoprotein (P-gp) inhibitors within 7 days prior to day 1 of protocol therapy.
* Narrow therapeutic index P-gp substrates within 7 days prior to day 1 of protocol therapy.
* Acute promyelocytic leukemia.
* Active central nervous system (CNS) leukemia.
* Active fungal infection or bacterial sepsis.
* Class III/IV cardiovascular disability according to the New York Heart Association classification.
* Participants with clinically significant arrhythmia or arrhythmias not stable on medical management within two weeks of enrollment. Subjects with controlled, asymptomatic atrial fibrillation can enroll.
* History of acute cardiovascular ischemic event, i.e., myocardial infarction or unstable angina within 6 months of enrollment.
* History of unexplained syncope, significant histories of CAD (requiring revascularization by percutaneous coronary intervention \[PCI\] or coronary artery bypass grafting \[CABG\]), cardiomyopathy (ejection fraction \[EF\] \< 50%).
* Prior surgery or gastrointestinal dysfunction that may affect drug absorption (e.g., gastric bypass surgery, gastrectomy).
* Unable to swallow capsules, has a partial or small bowel obstruction, or has a gastrointestinal condition resulting in a malabsorptive syndrome (e.g. small bowel resection with malabsorption).
* Active peptic ulcer disease.
* Other active malignancy except for localized skin cancer, bladder, prostate, breast or cervical carcinoma in situ.
* Females only: Pregnant or breastfeeding.
* Any other condition that would, in the Investigator's judgment, contraindicate the patient's participation in the clinical study due to safety concerns with clinical study procedures.
* Prospective participants who, in the opinion of the investigator, may not be able to comply with all study procedures (including compliance issues related to feasibility/logistics).
