Multiple-input multiple-output (MIMO) is a promising wireless communication technique to boost the wireless link capacity without requiring additional power or bandwidth. Combined with orthogonal frequency division multiplexing (OFDM) the MIMO OFDM system is considered an attractive candidate for high data rate wideband wireless communications over frequency selective channels. 4G-LTE, WiFi (802.11) and WiMax (802.16e) all adopt MIMO OFDM in their standards. ☐ In real world applications of MIMO OFDM system, diversified sources of interference often exist and significantly influence the system performance of MIMO OFDM. Firstly let's consider the single user point-to-point MIMO OFDM system. The interference may be often caused by a long time delay spread of the channel. Typically MIMO OFDM systems resort to cyclic prefix (CP) to protect the system from interference. The simple one-tap equalization in OFDM also relies on the assumption that the CP length is at least as large as the channel memory length. However, in a number of applications, the sufficient CP length requirement is not practically satisfied for various reasons. For example, in the next generation of WiFi standard, outdoor transmission is much more involved. The time delay spread of the outdoor channel may be much longer than that considered in the typical channel models in the current WiFi standards. If the current OFDM symbol structure is inherited, we may face the insufficient CP problem in the outdoor transmission scenario. Next consider the interference scenario in MIMO OFDM cooperative communication system. In the emerging 5G cooperative communication system, the relay node may adopt the amplify-and-forward (AF) relay protocol and the in-band full-duplex (FD) mode. The simultaneous transmission and reception at the relay node gives rise to severe self loopback interference. All the mentioned types of interference have detrimental influence on the error rate performance of the MIMO OFDM system. Therefore signal designs for MIMO OFDM system must take interference cancellation or management into consideration. ☐ In the first topic, a new interference nulling based channel independent precoding for MIMO OFDM systems of nt transmit and nr receive antennas with insufficient CP is proposed. When the CP length is insufficient, intercarrier interference (ICI) from the symbols in the current transmission block and interblock interference (IBI) from the previous transmission block occur in the OFDM system. To cope with ICI and IBI induced by the insufficient CP, we have proposed a design framework to completely nulling the IBI. In our first channel independent precoding scheme, no channel state information is needed. When MIMO OFDM system equips the same number of transmit and receive antennas or the transmit antennas are more than the receive antennas, we have shown that precoding at the transmitter combined with interference nulling at the receiver can assist to either cancel interblock interference (IBI) or separate the subspace occupied by IBI from the subspace used for information symbols. When the transmit antennas are less than the receive antennas, it is demonstrated that information symbols can be decoded without any precoding at the transmitter but just with interference nulling and block linear equalization at the receiver. ☐ In the next topic, we consider a robust precoder design which combines statistical CSI at the transmitter and the interference nulling based precoding structure to gain a better performance compared to the channel independent precoding scheme. Statistical channel state information at the transmitter in the form of covariance matrix of the MIMO OFDM channel matrix is utilized to perform the robust precoding. Standard optimization problem is formulated with regard to the largest mean square error (MSE) of the minimum mean square error (MMSE) equalized symbol in the insufficient CP signal model. By employing convex optimization technique, a closed-form solution is derived for the optimization problem. Since the average MSEs are set equal using this precoder, the BER performance is better than our previously proposed channel independent precoder. ☐ In the third topic, we consider the OFDM-based transmission in the AF full duplex relay system with residual self-interference. A delay diversity OFDM (DD OFDM) transmission scheme in amplify-and-forward (AF) full-duplex relay systems is investigated. One direct source-to-destination link, one relay forwarding link and residual self-interference (RSI) are considered in the system. The necessary cyclic prefix (CP) length is investigated and a suitable AF relay protocol in the full-duplex relay OFDM system is proposed. This paper demonstrates that the AF relay link and the direct source-to-destination link can be combined to provide spatial diversity. The key is that the DD OFDM scheme is used to transform the spatial diversity into increased channel frequency diversity that is further exploited by using the bit-interleaved coding. The BER performance of the proposed system is verified by simulation results.