There recently have been many satellite-based data products published that describe aboveground biomass (AGB) that often span similar areas, making it hard to decide which AGB product should be used to design nature-based climate solutions or studies of global ecology. High-quality reference data, such as field-based measurements, are usually lacking, so rigorous validation that support a decision about these datasets is likewise lacking. Yet we expect AGB should respond realistically to ecological processes - so perhaps we can evaluate AGB data products not on accuracy, but on their behavior in response to disturbance like wildfire or how they vary with climate gradients. Here, we ask three questions about a set of publicly available datasets - how disparate are their estimates in different ecosystems? which ones vary realistically in time with disturbance? which ones vary realistically in space with climate gradients? Initial work shows limited spread at the global level, but high disagreement at the state-level. Many datasets fail to lose and recover AGB in response to fire and fail to characterize non-forest or post-fire ecosystems. This framework can be used to rapidly evaluate the many expected new datasets coming from upcoming NASA missions like NISAR.