A defining characteristic of current U.S. educational policy is the use of data to inform decisions about resource allocation, teacher hiring, and curriculum and instruction. Perhaps the biggest challenge to data-driven decision making (DDDM) is that data use alone does not automatically result in improved teaching and learning. Research indicates that translating raw data into useable information and actionable knowledge for teachers requires not only adequate technical and social supports, but also an awareness of how educators in real-world settings actually use information to make decisions. Yet, little is known about DDDM in higher education, in general, and how postsecondary faculty make sense of and use data in their instructional decision-making processes, in particular. In this paper, we use naturalistic decision-making theory to generate practice-based descriptions of how 59 STEM faculty at three large public research universities used data as part of their course planning. Interview transcripts and notes taken while observing planning meetings were analyzed using an inductive approach to content analysis. In practice, respondents used different types of data and other information obtained from, for example, student assessments, end-of-semester evaluations, and conversations with colleagues. Results indicate that faculty generally collect and analyze data in informal, ad hoc scenarios ungoverned by institutional policy. Exceptions include disciplines with accreditation pressures and team-taught courses where structured (and supported) opportunities exist for faculty to collect, analyze, and reflect upon data about student learning. Thus, while numeric data are clearly viewed by this population of faculty as the most rigorous, in practice, even those that use quantitative data also use other sources of information. These results suggest an opportunity for educational leaders to design policies and professional development initiatives that facilitate a more formal collection of and reflection on data by faculty. In pursuing such technical solutions, however, policymakers and educational leaders must carefully negotiate the tension between rigor and relevance, and learn from the challenges experienced in the K-12 sector regarding DDDM.