"Our visual system is sensitive to boundaries defined by differences in cues such as luminance (first-order cue), as well as texture, contrast, or motion (second-order cues). Gradients in these cues can be utilized to perform tasks such as figure-ground segregation and 3D shape perception. A significant fraction of neurons in the early visual cortex of cats and monkeys have been shown to be selective to both first- and second-order boundaries. These neurons are thought to be the neural correlate for perceptual encoding of such boundaries. They are selective for the same boundary orientation irrespective of the cue (first- or second-order) that defines it ("form cue-invariance"), which makes these neurons powerful candidates for the task of segmentation. However, the neural circuitry that gives rise to this selectivity for the early stages of visual processing remains unclear. To address this question, I perform neurophysiological recordings at the early stages of the visual pathway in cats, and then build biologically inspired neural circuit models that can account for visual response properties of neurons at subcortical as well as early cortical stages. In Chapter 2, I use multi-electrode recordings to demonstrate the presence of a significant fraction of neurons in cat Area 18 with nonlinear receptive fields like those of subcortical Y-type cells. These neurons have receptive field properties intermediate between subcortical Y cells and cortical orientation selective cue-invariant neurons. These are strong candidates for building cue-invariant orientation-selective neurons. Furthermore I present a novel neural circuit model that pools such Y-like neurons in an unbalanced "push-pull" manner, to generate orientation-selective cue-invariant receptive fields.In Chapter 3, I estimate biologically constrained neural network models of cat LGN receptive fields using recent machine learning methods (deep learning). The receptive fields are modeled as arising from a two-stage convolutional neural network model. The first stage, corresponding to retinal bipolar cell subunits, is modeled as a convolutional filter layer, and the second stage is modeled as a pooling layer. These two layers are separated by an intermediate parametric nonlinearity. I train such a neural network model for each recorded LGN neuron, using its spiking responses to naturalistic texture stimuli. These models are not only better in comparison to the standard linear-nonlinear models at predicting response to arbitrary stimuli, but they also recover biologically interpretable subunit models.In chapter 4, I evaluate the integration of ON- and OFF-pathway inputs by individual neurons in early cortical areas of the cat (Area 17 and Area 18). In this study, I model receptive fields of cortical simple cells as a linear weighted sum of rectified inputs from model ON- and OFF-center LGN afferents, with the weights estimated using a regression framework. The estimated models reveal significant asymmetries in spatiotemporal integration of ON and OFF signals within simple cell receptive fields. These observed asymmetries could provide the neural mechanism for generating cue-invariant receptive fields from Y-pathway inputs.In summary, I put together our knowledge of retinal as well as early cortical processing to show how spatial nonlinearities emerging from the retina could provide an essential basis for cortical visual processing. I further evaluate these neural mechanisms by estimating single neuron receptive field models, using modern system identification methods. Finally I propose, and provide supportive evidence for, a novel neural circuit mechanism that could explain the cue-invariant processing of luminance- and texture-defined boundaries through a common pathway." --