Estimation of unknown quantities in a nonlinear dynamic system has been a challenge of great interest and importance, which is encountered in different research fields. Founded on a perspective of updating probabilistic belief on unknown quantities with observations, Bayesian analysis has provided a useful methodology and framework for construction of various estimation techniques. This dissertation presents a study of some new developments and applications of Bayesian estimation theory. Both filtering and smoothing will be considered --- the former concerns estimation of the present situation using measurements up until the present time, and the latter is about estimation of the past using all the measurements. In the dissertation, we investigate both state estimation and simultaneous input and state estimation. For the former, Bayesian filtering in a Gaussian context is discussed. We propose to use the radial basis function approximation as a desirable option to realize the Gaussian state filtering. We then improve the standard ensemble Kalman filter by introducing iterative optimization. Simultaneous input and state estimation has emerged as a new challenge. We extend the Bayesian methodology to deal with this problem and consider both filtering and smoothing cases. The Bayesian paradigms are built as a statistical foundation to fulfill this task, are built. On such a basis, we then develop a series of estimation methods using iterative optimization and Monte Carlo-based ensemble approaches. We further examine the link between our methods and the existing ones and analyze their properties especially in the linear case. The dissertation also studies application of estimation techniques to some real-world issues. We investigate real-time state-of-charge estimation for batteries, proposing an adaptive method based on multi-model state estimation. It allows for accurate estimation in the presence of uncertain or unknown variables and can promote the battery monitoring performance and operational safety potentially. The other application presented is oceanic flow field reconstruction. Flows exist everywhere in the ocean, playing a crucial role in many aspects of marine environment and biology. As part of a collaborative effort with Scripps Institution of Oceanography to build an original ocean observing system based on a group of drifters, we apply the simultaneous input and state estimation methods proposed to analyze the data collected by drifters to estimate the flow velocity and monitor the drifter's motion status. This work can be conducive to future endeavors in oceanographic research.