Stephen Crabb will take over as Work and Pensions Secretary after Iain Duncan Smith dramatically quit the Cabinet over cuts to disability benefits.
David Cameron has also moved Alun Cairns to the role of Welsh Secretary, replacing Mr Crabb, as he moved to shore up his front-bench team.
Mr Duncan Smith resigned from his government post on Friday as he launched an attack on the Budget.
He branded George Osborne's latest financial package as "indefensible".
In his letter of resignation, he said: "I have for some time and rather reluctantly come to believe that the latest changes to benefits to the disabled and the context in which they've been made are a compromise too far.
"While they are defensible in narrow terms, given the continuing deficit, they are not defensible in the way they were placed within a Budget that benefits higher earning taxpayers. They should have instead been part of a wider process to engage others in finding the best way to better focus resources on those most in need.
"I am unable to watch passively whilst certain policies are enacted in order to meet the fiscal self-imposed restraints that I believe are more and more perceived as distinctly political rather than in the national economic interest."
Mr Duncan Smith has been at loggerheads with Mr Cameron and Mr Osborne over whether Britain should stay in the EU, joining a handful of other Cabinet ministers in calling for Brexit.
But his letter to the Prime Minister indicated that the row over cuts to the Personal Independence Payment (PIP) had been the last straw.
In his letter responding to Mr Duncan Smith, Mr Cameron said he was "puzzled" by the decision.
Downing Street said Guto Bebb would take Mr Cairns' previous job as junior minister at the Wales Office and government whip, which was unpaid.
Read More
MORE ABOUT