The definition of the higher order difference frequencies differ for different signals. For SMPTE, DIN and AES-17 MD d3L = f2 - 2f1. For CCIF and AES-17 DFD d3L = 2f1 - f2 (the various frequencies are tabulated in
the help). For custom signals REW uses the DIN/SMPTE definition if f2/f1 is > 7, a somewhat arbitrary threshold, which isn't the case for the signal above. I could change that threshold of course, or use the amplitude ratio instead since modulation distortion signals are usually 4:1. Happy to hear suggestions.