This is the phase response as a result of including all room interactions. We typically correct only the direct sound response. If you are FIR EQing each channel separately then apply a heavy FDW filter to the each vector average. See what a 5 or 6 octave FDW looks like and if that is not enough to estimate the direct sound phase smoothly we can go as low as 2 octave, but that usually is not necessary and may be misleading in case where the XOs are not aligned well. All room ripples do not need to be removed. We can just ignore them and adjust the FIR phase correction filter to remove the overall smoothed shape. The 2 channels should be very similar for direct sound phase, but if they are heavily EQed independently they may deviate somewhat.
If you post an mdat containing the Left and Right vector averages (just the 2 measurements) I can take a look to see if it looks as expected.