Policymakers at the national level have recently initiated K-12 education reforms focused on teacher quality and teacher evaluation. Far-reaching legislation was subsequently enacted in the state of Arizona requiring schools to adopt standards-based teacher evaluation systems and link them to student outcomes. The end product is to result in annual summative measures of teacher effectiveness. Because of this, Arizona school administrators have become concerned about rapidly becoming experts in high-stakes teacher evaluation. Principals rarely have time to come together to talk about teacher evaluation, and consider the reliability of their evaluations and how to use teacher evaluation to help teachers improve their practice. This action research study focused on a group of nine administrators in a small urban district grappling with a more complex and high-stakes teacher evaluation system. An existing community of practice was engaged to help administrators become more effective, fair, and consistent evaluators. Activities were designed to engage the group in dynamic, contextualized learning. Participants interacted in small groups to interpret the meaning of newly adopted evaluation instruments and professional teaching standards, share practical knowledge, and compare teacher evaluation experiences in classrooms. Data were gathered with mixed methods. Prior to, and immediately after engaging in this 20-week innovation, principals and district administrators were given two surveys and interviewed about teacher evaluation. Additionally, a detailed record of this project was kept in the form of meeting records and a research journal. Quantitative and qualitative data were triangulated to validate findings. Results identified concerns and understandings of administrators as they attempted to come to a shared consensus regarding teacher evaluation, increase inter-rater reliability, and use teacher evaluation to improve professional practice. As a result of working and learning together administrators lowered their concerns about inter-rater reliability. Other concerns, however, remained and grew. Administrators found the process of gaining a common understanding of teacher evaluation to be complex and far more time-consuming than anticipated. Intense concerns about alignment of the evaluation system with other reforms led these administrators to consider modifications in their evaluation system. Implications from this study can be used to help other administrators grappling with common concerns.