When water evaporates from the ocean, not all water molecules evaporate at the same rate. Some of them are heavier than others and therefore need more time to reach the vapor phase. This transient effect is called nonequilibrium fractionation and contributes to a lower ratio between heavy and light water molecules in the atmosphere than in the ocean. In models, nonequilibrium fractionation is often expressed as a function of wind speed, following a formulation based on theory and wind tunnel experiments. However, many observational studies have questioned the validity of this formulation, in particular the discontinuity resulting from the transition between a smooth and rough wave regime. Here, we develop a new wind-speed dependent formulation for nonequilibrium fractionation that explicitly accounts for ocean waves, and test it in a climate model. We evaluate the climate model simulation with measurements of the isotope ratios of water vapor above the ocean surface from ships, and find that - in general - the new formulation improves the modeled ratio of heavy and light water molecules in vapor relative to the old formulation.