The need for computerized analysis of retinal images has been increasing with the wide clinical use of fundus photography. Retinopathy of prematurity (ROP) is among the diseases that can be diagnosed through the use of retinal images. It is a disease affecting low-birth weight infants, in which blood vessels in the retina of the eye develop abnormally and cause potential blindness. We propose an image processing and machine learning framework from the vasculature segmentation to diagnosis of Retinopathy of Prematurity (ROP) in retinal images. The system takes a retinal image as an input, does automatic vessel segmentation and tracing, extracts various features, performs feature selection and outputs a diagnostic decision. Although, ROP is the leading cause of childhood blindness in the world, there exists a wide variability among experts in diagnosis. We propose a method to do an in-depth feature and observer analysis by employing Mutual Information (MI) to understand the underlying causes of inter-expert disagreement. The contributions of this dissertation are (i) extraction of new features quantifying tortuosity and amount of branching that are useful for ROP diagnosis, (ii) a novel feature representation paradigm utilizing Gaussian Mixture Models of image features to better model tortuous and straight vessels, (iii) an accurate pairwise similarity measure between images based on the proposed feature representation, (iv) the use of the proposed similarity measure in support vector machine (SVM) and k-nearest neighbor (KNN) classifiers, and (v) a MI-based feature-observer analysis technique to understand the features that lead inter-expert disagreement. The proposed framework is the first fully-automated computer-aided diagnosis system for ROP disease. The experiments are carried out on two datasets each of which consists of wide-angle colored retinal images acquired during routine ROP exams. The first one is designed for feature-observer analysis and consists of 34 images diagnosed by 22 experts. The second one is designed for classification experiments and contains 77 images with reference standard diagnosis. In our feature-observer analysis, we observed that although ROP is defined based on arteriolar tortuosity and venous dilation, there exists other features that highly correlate with expert opinions. This finding shows that the definition of the disease is subjective. We obtain 95\% accuracy compared to the reference standard on the second dataset when we use features extracted from manual segmentations. This performance is comparable to the performance of the individual experts (96%, 93%, 92%), Williams test = 1.0. With the features extracted from computer-based segmentation algorithm, we achieve 80% accuracy, which is promising for a fully-automated system.