Victory, achieved with more than a day to spare, also saw England take the four-match series 3-1, with Joe Root triumphant in his first series as England captain.
South Africa, set a massive 380 to win, were dismissed for 202 shortly after tea on the fourth day, with Ali wrapping things up with two wickets in two balls.
That gave the off-spinner a return of five for 69 in 19.5 overs and 25 wickets in total in a series where he also scored 252 runs.
South Africa, 43 for three at lunch, saw Hashim Amla (83) and captain Faf du Plessis (61) put on 123 for the fourth wicket.
But Ali then took three wickets for five runs in 11 balls — including the prize scalp of Amla — as South Africa collapsed from 163 for three to 173 for six.
England had laid the platform for victory by establishing a first-innings lead of 136 that owed much to Jonny Bairstow’s 99 and a return of four for 38 from James Anderson — bowling from the newly-named James Anderson End at his Lancashire home ground.
Ali’s second-innings 75 not out then left South Africa with a huge chase.
England 362 (J Bairstow 99, B Stokes 58, J Root 52; K Rabada 4-91) and 362 and 243 (M Ali 75 no; M Morkel 4-41, D Olivier 3-38) v South Africa 226 (J Anderson 4-38, S Broad 3-46) and 202 (H Amla 83, F du Plessis 61; M Ali 5-69, J Anderson 3-16)
Result: England won by 177 runs
Series: England win four-match series 3-1