?
Might be worth trying to re foam the original speakers. If you get lucky maybe you can get some foams that will clear the rim around the front plate so that the foams don't rub up against the opening in the front plate. You will need a 5 inch foam with an angled lip that attaches to the speaker cone.
This is part of the reason the speakers are so stiff it's to control cone movement to avoid over excursion of the speaker cone. It's trail and error all you can do is try.
With the covers off I can reach through the port and hold my finger on the back of the cone. The distortion stops, but there's still more bass than before. So if I could add a bit of stiffness I might find a middle ground.
As stupid as it sounds you might try adding a bit of poly fill to the back of the speaker ports . this will add resistance to the cone movement by controlling the airflow through the ports.
One other idea is to switch over to a rubber surround. The rubber surround material is much heavier and more dense than foam and by its nature more resistance than foam but not too much or too little.
I said distort because I couldn't think of a better word to describe it. It's a rattle. The sound was nice and clear before I swapped the surrounds.
