Ministers in the UK are accountable to the
Prime Minister, who appoints them, and may remove them at any time (Exceptionally, the Queen might be able to block such a move - whether she would be successful is anyone's guess), to Parliament, and to their constituency, if a member of the Commons.
MPs must also have an eye to the mood of their party, as they normally intend to return to government once their
PM has bid high office farewell.
The constituency
The voters of the MP's constituency may fail to return them at their next election. Losing the support of their
constituency party will most certainly not remove an MP from any position whatsoever, although it may make re-election more difficult, as they will be standing as an independant from the next election.
Although Parliament cannot remove a minister from their position, except by the passage of an Act, which would be unprecedented, it can still subject a minister to censure. The most common step would be to require a full explanation before either a
select committee, or their House, of the reasons for any given decision. Either house may also pass motions indicating a lack of confidence in the minister, even asking the Queen and her Prime Minister to consider the minister's position. In the ultimate case, and this has never been done to anyone in generations, the Commons may summon anyone to the bar of the house (or to the house, if an MP), and try and imprison them.
All change!
Once the election has come and a new party installed, they have the advantage of a
civil service, so the transition of power is almost entirely seamless. Any
manifesto commitment to invade Gondwanaland can be actioned immediately (to the extent that an army can ever be immobilised immediately), as the new ministers have little need to figure out the email system.
The disadvantage of a civil service is that it is perceived as a large body of very great inertia, acting to curb the propensity of ministers to do anything that might upset their world, or do anything that the civil service does not approve of.