Stop Mail Changing Port for E-Mail Account

I'm having an annoying problem with Mail on Yosemite, which I didn't notice on earlier versions. Basically if I suffer a connection issue with my ISP's mail service (which does sometimes become slow to respond during peak hours) then Mail seems to try alternative ports.

This would be fine, except that it eventually settles on port 143 with SSL disabled, at which point it refuses to connect (because it can't send my password securely), but it has now changed the account's port permanently to 143, meaning I have to change it back before I can use my e-mail account again.

Is there a way to prevent Mail from doing this? I've set the correct port for the account, and know it's correct, so I don't want Mail to ever change it, but it keeps doing this any time the connection takes a bit too long to respond.