Driven by the significant consumer demand for reliable and high data rate communications, the future-generation cellular systems are expected to employ cutting-edge techniques to improve the service provisioning at substantially reduced costs. Cooperative relaying is one of the primary techniques due to its ability to improve the spectrum utilization by taking advantage of the broadcast nature of wireless signals. This dissertation studies the physical layer cooperative relaying technique and resource allocation schemes in the cooperative cellular networks to improve the spectrum and energy efficiency from the perspectives of downlink transmission, uplink transmission and device-to-device transmission, respectively. For the downlink transmission, we consider an LTE-Advanced cooperative cellular network with the deployment of Type II in-band decode-and-forward relay stations (RSs) to enhance the cell-edge throughput and to extend the coverage area. This type of relays can better exploit the broadcast nature of wireless signals while improving the utilization of existing allocated spectral resources. For such a network, we propose joint orthogonal frequency division multiplexing (OFDM) subcarrier and power allocation schemes to optimize the downlink multi-user transmission efficiency. Firstly, an optimal power dividing method between eNB and RS is proposed to maximize the achievable rate on each subcarrier. Based on this result, we show that the optimal joint resource allocation scheme for maximizing the overall throughput is to allocate each subcarrier to the user with the best channel quality and to distribute power in a water-filling manner. Since the users' Quality of Service (QoS) provision is one of the major design objectives in cellular networks, we further formulate a lexicographical optimization problem to maximize the minimum rate of all users while improving the overall throughput. A sufficient condition for optimality is derived. Due to the complexity of searching for the optimal solution, we then propose an efficient, low-complexity suboptimal joint resource allocation algorithm, which outperforms the existing suboptimal algorithms that simplify the joint design into separate allocation. Both theoretical and numerical analyses demonstrate that our proposed scheme can drastically improve the fairness as well as the overall throughput. As the physical layer uplink transmission technology for LTE-Advanced cellular network is based on single carrier frequency division multiple access (SC-FDMA) with frequency domain equalization (FDE), this dissertation further studies the uplink achievable rate and power allocation to improve the uplink spectrum efficiency in the cellular network. Different from the downlink OFDM system, signals on all subcarriers in the SC-FDMA system are transmitted sequentially rather than in parallel, thus the user's achievable rate is not simply the summation of the rates on all allocated subcarriers. Moreover, each user equipment (UE) has its own transmission power constraint instead of a total power constraint at the base station in the downlink case. Therefore, the uplink resource allocation problem in the LTE-Advanced system is more challenging. To this end, we first derive the achievable rates of the SC-FDMA system with two commonly-used FDE techniques, zero-forcing (ZF) equalization and minimum mean square error (MMSE) equalization, based on the joint superposition coding for cooperative relaying. We then propose optimal power allocation schemes among subcarriers at both UE and RS to maximize the overall throughput of the system. Theoretical analysis and numerical results are provided to demonstrate a significant gain in the system throughput by our proposed power allocation schemes. Besides the physical layer technology, the trend of improving energy efficiency in future cellular networks also motivates the network operators to continuously bring improvements in the entire network infrastructure. Such techniques include efficient base station (BS) redesign, opportunistic transmission such as device-to-device and cognitive radio communications. In the third part of this dissertation, we explore the potentials of employing cooperative relaying in a green device-to-device communication underlaying cellular network to improve the energy efficiency and spectrum utilization of the system. As the green base station is powered by sustainable energy, the design objective is to enhance both sustainability and efficiency of the device-to-device communication. Specifically, we first propose optimal power adaptation schemes to maximize the network spectrum efficiency under two practical power constraints. We then take the dynamics of the charging and discharging processes of the energy buffer at the BS into consideration to ensure the network sustainability. To this end, the energy buffer is modeled as a G/D/1 queue where the input energy has a general distribution. Power allocation schemes are proposed based on the statistics of the energy buffer to further enhance the network efficiency and sustainability. Theoretical analysis and numerical results are presented to demonstrate that our proposed power allocation schemes can improve the network throughput while maintaining the network sustainability at a certain level. Our analyses developed in this dissertation indicate that the cooperative transmission based on cooperative relaying can significantly improve the spectrum efficiency and energy efficiency of the cellular network for downlink transmission, uplink transmission and device-to-device communication. Our proposed cooperative relaying technique and resource allocation schemes can provide efficient solutions to practical design and optimization of future-generation cellular networks.