Spin orbit coupling (SOC) is a relativistic phenomenon that couples the spin angular momentum of an electron to its orbital angular momentum. This leads to remarkable consequences for the motion of electrons in a crystalline solid that are not only magnified but also qualitatively different when electrons interact strongly with each other. As one goes down the periodic table, the strength of SOC usually increases due to the larger positive charge of the nucleus that produces an enhanced magnetic field felt by electrons. On the other hand, electron–electron correlations, characterized by on–site Coulomb repulsion U, decrease due to the larger extent of the orbitals. Hence a perfect playground to study the competition between correlations and SOC are the transition metal compounds with partially filled 4d and 5d orbitals. My research focuses how this interplay between strong correlations and SOC gives rise to unconventional magnetic structures with unusual properties, where both the orbital and spin orderings dictate the nature of magnetism. Since orbitals are directional and couple to the lattice, magnetism in these materials can be manipulated by external strain, providing a new knob to tune their magnetic properties. Specifically, I obtained phase diagrams of cubic 5d double perovskite Mott insulators in which the transition metal ion is surrounded by an octahedral cage of oxygen atoms. My calculations performed at a mean-field level are able to explain long-standing experimental puzzles related to missing entropy and unusual magnetic susceptibilities in these compounds. In addition, I also construct low energy effective Hamiltonian for distorted 5d 1 double perovskite Mott insulators and find AFM ground states with mean–field theory, which matches experimental results. A new design principle for topological flat bands utilizing strong spin–orbit coupling and orbital frustration is proposed on a honeycomb lattice of transition metal ions.