We report the surface reaction mechanism of the atomic layer deposition (ALD) of niobium oxide films by combining experimental and theoretical studies. Tert-butylimido tris(dimethylamido)niobium [tBuN=Nb(NMe2)3] and ozone (O3) were studied as the precursor and co-reactant. The in situ quartz crystal microbalance (QCM) analysis predicted that two HNMe2 molecules or one H2NtBu molecule were released at 200 °C per one molecule tBuN=Nb(NMe2)3. Since the QCM alone could not distinguish two possible reaction pathways, we compared two pathways by density functional theory (DFT) calculations. We simulated the chemisorption of the precursor molecule on the –OH terminated T-Nb2O5 substrate. The release of two –NMe2 ligands was favorable due to low activation energies of 0.71 and 0.63 eV for the first and second releases. On the other hand, the release of =NtBu was not favorable due to the high activation energy of 1.50 eV. Therefore, the amido and imido ligands were actor and spectator ligands during the first half-reaction of the ALD of niobium oxide films. The combination of QCM and DFT could successfully reveal the surface reaction mechanism of the ALD process using a heteroleptic precursor.