The transition metal dichalcogenides are layered material and have been successfully exfoliated to atomically thin layers in experiment. Recently extensive study are focused on the atomically thin transition metal dichalcogenides due to the various novel properties the material exhibits. In the monolayer 2H-transition metal dichaocogenides, the intrinsic inversion symmetry breaking in the crystal structure generates the strong spin-orbit coupling. As the crystal structure has the out of plane mirror symmetry, the electrons’ spin are all pinned to the z direction, forming the Ising type spin texture in the momentum space. As the Ising spin-orbit coupling dramatically modifies the electronic properties, especially, it strongly influences the superconducting 2H type atomically thin transition metal dichalcogenides. In addition to the 2H type, 1T type transition metal dichalcogenides maintain its inversion symmetry and have different crystal structure, so the physical properties of 1T-TaS2 are different from its 2H counterpart. In this thesis, we will focus on the effect of Ising spin-orbit coupling on the 2H superconducting transition metal dichalcogenides of NbSe2 and MoS2, and study the Mott physics in 1T-TaS2. As the spin-orbit coupling plays the key role in the novel properties shown in transition metal dichalcogenides, at last we study the novel topological phase in the spin-orbit coupled ultra cold atoms in the cubic optical lattice.

In the chapter 2, 3, we focus on the superconducting atomic NbSe2 layers and study the effect of in-plane magnetic field on the superconducting properties. We will show that the Ising spin-orbit coupling strongly enhance in-plane upper critical field in the 2D NbSe2. In the low temperature, the in-plane magnetic field induced continuous phase transition is observed from the superconductivity to the normal metal, unlike the abrupt first order transition in the conventional BCS superconductor. Since the Ising enhanced in-plane Hc2 goes much beyond the Pauli limit HP , we further study the superconductivity state in the presence of in-plane magnetic field between Hc2 and HP . We found the in-plane magnetic field can drive the superconducting monolayer NbSe2 to a nodal topological superconductor, with a large number of Majorana zero modes at the edge forming the Majorana flat bands connecting the nodes. The tunnelling spectroscopy is proposed to detect the topological phase transition.

In the chapter 4, we discuss the superconductivity pairing symmetry in the strong gated MoS2. A recent tunnelling spectroscopy shows the density of states for strong gated superconducting MoS2 has the “V ” shape instead of the standard “U” shape. Combined with the group theory, we systematically analyze the possible pairing in MoS2 and show that a spin-triplet nodal pairing with the d vector parallel to the Ising spin-orbit coupling can consistently explain the experimental tunnelling spectroscopy measurement. In order to further confirm the spin-triplet pairing order parameter, the phase sensitive measurement is further suggested to directly measure the parity of the superconductivity.

In the chapter 5, we study the Mott insulating state in the 1T-TaS2. The 1T-TaS2 is a cluster Mott insulator with 13 Ta atoms forming the star of David as the unit cell. The first principle calculation shows a narrow band is isolated from other bands in the star of David superlattice. As the isolated band is half filled, the Mott insulating state is described by the half filled Hubbard model. We derive the effective spin model up to the ring exchagne term from the Hubbard model and use the density matrix renormalization group to numerically solve the spin model. There is no spin order observed but a spinon Fermi surface emerges in the spin-spin correlation function. We conclude that the 1T-TaS2 is possible a gapless spin liquid candidate with spinon Fermi surface.

In the chapter 6, we systematically study the superfluid phase in the spin-orbit coupled cold atoms in the cubic optical lattice. In the presence of the 1D synthetic spin-orbit coupling, the nodal ring superfluid emerges with a drumhead like Majorana pockets at the surface. With 2D spin-orbit
coupling, most part of the nodal ring degeneracies are gapped out and only point nodes are left, the nodal ring superfluid evolves into the Weyl superfluid with Majorana flat bands connecting the Weyl nodes. In the 3D spin-orbit coupling, the Weyl nodes in the Weyl superfluid will have
energy shift so that the Majorana flat bands gain finite slope. It gives the Majorana zero modes at the surface finite group velocity and makes Majorana zero modes spiral forward at the surface. The large numbers of Majorana zero modes at the surface are suggested to use spatial resolved radio-frequency spectroscopy to detect.

Appendix A, B, C, D, E contain the details of the theoretical derivation and numerically calculation for the previous chapters.