Purpose: Image quality modeling is important in the development and optimization of imaging systems. In this work, detectability index (d') was derived from a task‐based 3D cascaded systems model for tomosynthesis and cone‐beam CT (CBCT). The extent to which detectability index presents a meaningful performance metric was assessed by direct comparison with human observer response over a broad range of imaging conditions and tasks. Methods: Detectability index was generalized to include anatomical background as an additional power‐law noise source. Four observer models were investigated: pre‐whitening, nonprewhitening, and variations thereof with eye filter and internal noise. A phantom was designed to present power‐law noise, providing an experimental basis for observer studies. Imaging tasks included a range of detection and discrimination tasks in both uniform and power‐law backgrounds. Human observer performance was measured from 9AFC tests, and directly compared with theoretical d' in terms of area under the ROC curve, Az, as a function of imaging task and orbital extent (qtot). Results: Theoretical d' demonstrated reasonable correspondence with real observers for all tasks and orbital extents. Typical cases showed increasing Az with qtot due to rejection of out‐of‐plane clutter, while constant‐dose cases showed that Az could decline with qtot due to increased quantum noise and view aliasing. Prewhitening models tended to overestimate observer performance, whereas non‐prewhitening models yielded fair agreement. While not specific to any particular application, the results quantitatively identify system design and technique considerations for applications in breast and chest tomosynthesis and CBCT. Conclusions: Generalized detectability index was validated with real observer performance across a wide range of imaging conditions and tasks, thereby providing a meaningful metric based on prevalent Fourier‐based characterization of imaging performance. The results identify a framework for optimizing tomosynthesis and CBCT performance and begin to bridge the gap between Fourier‐ and observer‐based image quality characterization.