In this thesis, I describe work aimed at understanding nonlinear material responses initiated by strong terahertz (THz) field excitation. I discuss two aspects of nonlinear THz spectroscopy in condensed-matter materials: developments of experimental THz capabilities and spectroscopy methods and their applications in investigating ultrafast nonlinear dynamics in different classes of materials. I first describe the THz generation, detection and spectroscopy methods, which are the basis of all of our studies. We have generated strong single- and multi-cycle THz pulses covering several spectral ranges using inorganic and organic crystals and developed linear and nonlinear THz spectroscopy techniques to interrogate light-matter interactions based on different observables and/or symmetry criteria. We have demonstrated a new method for studying time-domain electron paramagnetic resonance that allows us to measure THz-frequency fine structures of spin energy levels on a tabletop and have developed nonlinear two-dimensional (2D) magnetic resonance spectroscopy to distinguish nonlinear THz-spin interaction pathways. We also show that THz-pump, optical-probe spectroscopy, including THz field-induced second-harmonic generation spectroscopy and THz Kerr effect spectroscopy, can be extended to study phase transitions in quantum paraelectric and topological materials. We have employed the THz methods to drive and detect nonlinear responses from several degrees of freedom in the materials. We have demonstrated collective coherent control over material structure by inducing a quantum paraelectric to ferroelectric phase transition using intense THz electric fields in strontium titanate. We show that a single-cycle THz field is able to drive ions along the microscopic pathway leading directly to their locations in a new crystalline phase on an ultrafast timescale. We have driven highly nonlinear lattice and electronic responses in a topological crystalline insulator by dynamically perturbing the protecting crystalline symmetry through THz phonon excitation. We have observed oscillations in optical reflectivity that may be associated with electronic gap opening and modulation in the topological surface states. Finally, we have demonstrated nonlinear manipulation of collective spin waves in a canted antiferromagnet using strong THz magnetic fields and we have observed full sets of the second- and third-order nonlinear responses in 2D THz magnetic resonance spectra, which are accurately reproduced in our numerical simulations.