Publications

2024

1. AABI

   Towards Model-Agnostic Posterior Approximation for Fast and Accurate Variational Autoencoders

   Y Yacoby,  W Pan,  and F Doshi-Velez

   Accepted @ Workshop at AABI 2024

   Abs PDF

   Inference for Variational Autoencoders (VAEs) consists of learning two models: (1) a generative model, which transforms a simple distribution over a latent space into the distribution over observed data, and (2) an inference model, which approximates the posterior of the latent codes given data. The two components are learned jointly via a lower bound to the generative model’s log marginal likelihood. In early phases of joint training, the inference model poorly approximates the latent code posteriors. Recent work showed that this leads optimization to get stuck in local optima, negatively impacting the learned generative model. As such, recent work suggests ensuring a high-quality inference model via iterative training: maximizing the objective function relative to the inference model before every update to the generative model. Unfortunately, iterative training is inefficient, requiring heuristic criteria for reverting from iterative to joint training for speed. Here, we suggest an inference method that trains the generative and inference models independently. It approximates the posterior of the true model a priori; fixing this posterior approximation, we then maximize the lower bound relative to only the generative model. By conventional wisdom, this approach should rely on the true prior and likelihood of the true model to approximate its posterior (which are unknown). However, we show that we can compute a deterministic, model-agnostic posterior approximation (MAPA) of the true model’s posterior. We then use MAPA to develop a proof-of-concept inference method. We present preliminary results on low-dimensional synthetic data that (1) MAPA captures the trend of the true posterior, and (2) our MAPA-based inference performs better density estimation with less computation than baselines. Lastly, we present a roadmap for scaling the MAPA-based inference method to high-dimensional data.

2. Nature

   Building personalized machine learning models using real-time monitoring data to predict idiographic suicidal thoughts

   S Wang,  R Genugten,  Y Yacoby,  W Pan,  K Bentley,  S Bird,  R Buonopane,  A Christie,  M Daniel,  D DeMarco,  A Haim,  L Follet,  R Fortgang,  F Kelly,  E Kleiman,  A Millner,  O Obi-Obasi,  J Onnela,  N Ramlal,  J Ricard,  J Smoller,  T Tambedou,  K Zuromski,  and M Nock

   Full paper on Nature Mental Health 2024

   Abs PDF

   Suicide risk is highest immediately after psychiatric hospitalization, but the field lacks methods for identifying which patients are at greatest risk, and when. We built personalized models predicting suicidal thoughts after psychiatric hospital visits (N=89 patients), using ecological momentary assessment (EMA; average EMA responses per participant=311). We built several idiographic models, including baseline autoregressive and elastic net models (using single train/test split) and Gaussian Process (GP) models (using an iterative rolling-forward prediction method). Simple GP models provided the best prediction of suicidal urges (R2_average of 0.17), outperforming baseline autoregressive (R2_average of 0.10) and elastic net (R2_average of 0.07) models. Similarly, simple GP models provided the best prediction of suicidal intent (R2_average of 0.12) compared to autoregressive (R2_average of 0.08) and elastic net (R2_average of 0.06). Findings suggest idiographic prediction of suicidal thoughts is possible, though accuracy currently is modest. Building GP models that iteratively update and learn symptom dynamics over time could provide important information to inform development of just-in-time adaptive interventions.

2023

1. SIGCSE

   Empowering First-Year Computer Science Ph.D. Students to Create a Culture that Values Community and Mental Health

   Y Yacoby,  J Girash,  and D Parkes

   Accepted @ SIGCSE 2023 Oral Presentation

   Abs PDF

   Doctoral programs often have high rates of depression, anxiety, isolation, and imposter phenomenon. Consequently, graduating students may feel inadequately prepared for research-focused careers, contributing to an attrition of talent. Prior work identifies an important contributing factor to maladjustment: that, even with prior exposure to research, entering Ph.D. students often have problematically idealized views of science. These preconceptions can become obstacles for students in their own professional growth. Unfortunately, existing curricular and extracurricular programming in many doctoral programs do not include mechanisms to systematically address students’ misconceptions of their profession. In this work, we describe a new initiative at our institution that aims to address Ph.D. mental health via a mandatory seminar for entering doctoral students. The seminar is designed to build professional resilience in students by (1) increasing self-regulatory competence, and (2) teaching students to proactively examine academic cultural values, and to participate in shaping them. Our evaluation indicates that students improved in both areas after completing the seminar.

2022

1. ICBINB

   An Empirical Analysis of the Advantages of Finite- vs. Infinite-Width Bayesian Neural Networks

   J Yao,  Y Yacoby,  B Coker,  W Pan,  and F Doshi-Velez

   Accepted @ NeurIPS ICBINB 2022

   Abs PDF

   Comparing Bayesian neural networks (BNNs) with different widths is challenging because, as the width increases, multiple model properties change simultaneously, and, inference in the finite-width case is intractable. In this work, we empirically compare finite- and infinite-width BNNs, and provide quantitative and qualitative explanations for their performance difference. We find that when the model is mis-specified, increasing width can hurt BNN performance. In these cases, we provide evidence that finite-width BNNs generalize better partially due to the properties of their frequency spectrum that allows them to adapt under model mismatch.

2. JMLR ICML UDL

   Mitigating the Effects of Non-Identifiability on Inference for Bayesian Neural Networks with Latent Variables

   *Y Yacoby,  *W Pan,  and F Doshi-Velez

   Accepted @ JMLR 2022
   Previous version accepted @ ICML UDL 2019 Spotlight Talk

   Abs PDF

   Bayesian Neural Networks with Latent Variables (BNN+LVs) capture predictive uncertainty by explicitly modeling model uncertainty (via priors on network weights) and environmental stochasticity (via a latent input noise variable). In this work, we first show that BNN+LV suffers from a serious form of non-identifiability: explanatory power can be transferred between the model parameters and latent variables while fitting the data equally well. We demonstrate that as a result, in the limit of infinite data, the posterior mode over the network weights and latent variables is asymptotically biased away from the ground-truth. Due to this asymptotic bias, traditional inference methods may in practice yield parameters that generalize poorly and misestimate uncertainty. Next, we develop a novel inference procedure that explicitly mitigates the effects of likelihood non-identifiability during training and yields high-quality predictions as well as uncertainty estimates. We demonstrate that our inference method improves upon benchmark methods across a range of synthetic and real data-sets.

3. HCOMP CHI HCXAI

   "If it didn’t happen, why would I change my decision?": How Judges Respond to Counterfactual Explanations for the Public Safety Assessment

   Y Yacoby,  B Green,  C Griffin,  and F Doshi-Velez

   Accepted @ HCOMP 2022
   Previous version accepted @ CHI HCXAI 2022 Oral Presentation

   Abs PDF

   Many researchers and policymakers have expressed excitement about algorithmic explanations enabling more fair and responsible decision-making. However, recent experimental studies have found that explanations do not always improve human use of algorithmic advice. In this study, we shed light on how people interpret and respond to counterfactual explanations (CFEs) – explanations that show how a model’s output would change with marginal changes to its input(s) – in the context of pretrial risk assessment instruments (PRAIs). We ran think-aloud trials with eight sitting U.S. state court judges, providing them with recommendations from a PRAI that includes CFEs. We found that the CFEs did not alter the judges’ decisions. At first, judges misinterpreted the counterfactuals as real – rather than hypothetical – changes to defendants. Once judges understood what the counterfactuals meant, they ignored them, stating their role is only to make decisions regarding the actual defendant in question. The judges also expressed a mix of reasons for ignoring or following the advice of the PRAI without CFEs. These results add to the literature detailing the unexpected ways in which people respond to algorithms and explanations. They also highlight new challenges associated with improving human-algorithm collaborations through explanations.

4. arXiv ICML UDL

   Failure Modes of Variational Autoencoders and Their Effects on Downstream Tasks

   Y Yacoby,  W Pan,  and F Doshi-Velez

   Full paper on arXiv 2022
   Previous version accepted @ ICML UDL 2020

   Abs PDF

   Variational Auto-encoders (VAEs) are deep generative latent variable models that are widely used for a number of downstream tasks. While it has been demonstrated that VAE training can suffer from a number of pathologies, existing literature lacks characterizations of exactly when these pathologies occur and how they impact downstream task performance. In this paper, we concretely characterize conditions under which VAE training exhibits pathologies and connect these failure modes to undesirable effects on specific downstream tasks, such as learning compressed and disentangled representations, adversarial robustness, and semi-supervised learning.

5. arXiv ICML UDL

   Uncertainty-Aware (UNA) Bases for Deep Bayesian Regression Using Multi-Headed Auxiliary Networks

   *S Thakur,  *C Lorsung,  *Y Yacoby,  F Doshi-Velez,  and W Pan

   Full paper on arXiv 2022
   Previous version accepted @ ICML UDL 2020

   Abs PDF

   Neural Linear Models (NLM) are deep Bayesian models that produce predictive uncertainties by learning features from the data and then performing Bayesian linear regression over these features. Despite their popularity, few works have focused on methodically evaluating the predictive uncertainties of these models. In this work, we demonstrate that traditional training procedures for NLMs drastically underestimate uncertainty on out-of-distribution inputs, and that they therefore cannot be naively deployed in risk-sensitive applications. We identify the underlying reasons for this behavior and propose a novel training framework that captures useful predictive uncertainties for downstream tasks.

2020

1. ICML UDL

   BACOUN: Bayesian Classifiers with Out-of-Distribution Uncertainty

   T Guénais,  D Vamvourellis,  Y Yacoby,  F Doshi-Velez,  and W Pan

   Accepted @ ICML UDL 2020

   Abs PDF

   Traditional training of deep classifiers yields overconfident models that are not reliable under dataset shift. We propose a Bayesian framework to obtain reliable uncertainty estimates for deep classifiers. Our approach consists of a plug-in "generator" used to augment the data with an additional class of points that lie on the boundary of the training data, followed by Bayesian inference on top of features that are trained to distinguish these "out-of-distribution" points.

2. ICML WHI

   CRUDS: Counterfactual Recourse using Disentangled Subspaces

   M Downs,  J Chu,  Y Yacoby,  F Doshi-Velez,  and W Pan

   Accepted @ ICML WHI 2020

   Abs PDF

   Algorithmic recourse is the task of generating a set of actions that will allow individuals to achieve a more favorable outcome under a given algorithmic decision system. Using the Conditional Subspace Variational Autoencoder (CSVAE), we propose a novel algorithmic recourse generation method, CRUDS, that generates multiple recourse satisfying underlying structure of the data as well as end-user specified constraints. We evaluate our method qualitatively and quantitatively on several synthetic and real datasets, demonstrating that CRUDS proposes recourse that are more realistic and actionable than baselines.

2019

1. AABI PMLR

   Characterizing and Avoiding Problematic Global Optima of Variational Autoencoders

   Y Yacoby,  W Pan,  and F Doshi-Velez

   Accepted @ AABI 2019 Spotlight Talk

   Additionally selected for publication @ PMLR 2019 Top 33%

   Abs PDF

   Variational Auto-encoders (VAEs) are deep generative latent variable models consisting of two components: a generative model that captures a data distribution p(x) by transforming a distribution p(z) over latent space, and an inference model that infers likely latent codes for each data point. Recent work shows that traditional training methods tend to yield solutions that violate modeling desiderata: (1) the learned generative model captures the observed data distribution but does so while ignoring the latent codes, resulting in codes that do not represent the data; (2) the aggregate of the learned latent codes does not match the prior p(z). This mismatch means that the learned generative model will be unable to generate realistic data with samples from p(z). In this paper, we demonstrate that both issues stem from the fact that the global optima of the VAE training objective often correspond to undesirable solutions. Our analysis builds on two observations: (1) the generative model is unidentifiable - there exist many generative models that explain the data equally well, each with different (and potentially unwanted) properties and (2) bias in the VAE objective - the VAE objective may prefer generative models that explain the data poorly but have posteriors that are easy to approximate. We present a novel inference method, LiBI, mitigating the problems identified in our analysis. On synthetic datasets, we show that LiBI can learn generative models that capture the data distribution and inference models that better satisfy modeling assumptions when traditional methods struggle to do so.

2. ASRM

   The application of machine learning methods to evaluate predictors of live birth in programmed thaw cycles

   D Vaughan,  W Pan,  Y Yacoby,  E Seidler,  A Leung,  F Doshi-Velez,  and D Sakkas

   Accepted @ Fertility and Sterility 2019

   PDF