Cooperative communication in relay networks facilitates high spectral efficiency and large wireless transmission coverage. Despite the advances of improved cooperative diversity and flexible node deployment, the relay network has more transmission nodes exposed in wireless environment than conventional single-hop channels and thus the secure communication may be impeded by potential eavesdroppers. Unlike the traditional approach of cryptographic design in the application layer, in this thesis we investigate the security issues in relay networks from a physical layer perspective. We start with a two-hop multiple-input multiple-output (MIMO) relay network and propose cooperative beamforming and jamming strategies to maximize the secrecy rate. The corresponding power allocation is also optimized via geometric programming. Both cases with known and unknown eavesdropper's channel state information (ECSI) are considered. Next, we study a case where the relay is untrusted although it helps with forwarding message to the destination. The exact secrecy outage probability (SOP) is characterized with the impact of the number of antennas at the relay. Antenna selection, beamforming and node selection schemes are proposed to suppress the wiretapping ability of the relay. Then we consider using a buffer-aided relay that can dynamically control its receive and transmit phases. A link selection scheme that takes into account both the two-hop transmission efficiency and security constraints is proposed and shown to significantly improve the secrecy throughput. Finally, we study robust transmit design when the relay acts as a pure cooperative jammer. The ECSI is assumed to be partially known and norm-bounded. The transmit covariance matrices are optimized via semidefinite programming, based on maximizing the worst-case secrecy rate. Through this thesis, we show that by taking advantage of the features of relay networks, the security performance can be significantly improved in terms of secrecy rate and SOP.