@Sonnie, I don't think that there is anything wrong with your listening session or conclusions.
The 328Bes are rear ported and passive and do not go particularly deep, presumably designed for efficiency and to paired with a subwoofer. A rear port can benefit by placement near walls, again not...