Having ascertained that you cannot downsize to below 19's you have a very limited size options available.
Using standard tyre data the 19" will have just 7mm more wall height than the 20" at 96mm Vs 89mm. This will make next to zero difference on speed bumps, which you'll have to treat with exactly the same respect. You’ll not turn it into a noticeably smoother ride, which I’m confident of having driven Jaguars and Range Rovers for years on a variety of rim sizes. Not that it will make any real difference but your actual speed will be 1.4% less in any given engine speed and of course narrower tyres and you‘ll need to inform your insurer due to the change from oe spec.
Overall given the aesthetic deterioration, lack of significant benefit and the cost of say c£1k for tyres and £1k for rims, I cannot see the justifiable benefit in doing it, but of course your call.
I'm on 20's and would never consider the change unless I was looking to a set of Winter wheels and tyres when the 19's would start to make sense (easier clean wheels, winter tread and compound, etc.), but certainly not needed in SE London