Waseem Isaacs have given the visitors the lead in the 34th minute but Phunya Sele Sele hit back with goals by Maloisane Ramasimong and Siphelele Luthuli in the 61st and 63rd minutes respectively.
After a rather cagey start to the game by both teams, it was Stellies that looked dangerous, especially from set-pieces.
Morgan Gould had a header tipped over the bar by Celtic goalkeeper Jackson Mabokgwane while Robyn Johannes was also denied after getting his head to the ball from a corner.
The newly-promoted side took the lead thanks to a howler from Mabokgwane as Isaacs struck a low shot straight at the shot-stopper but the keeper let the ball slip between his legs and into the net.
The visitors held onto their advantage until the halftime interval but the home side came out with renewed vigour in the second stanza.
And it was not long after that Celtic drew level as Ramasimong cushioned a header past Boy de Jong in the Stellenbosch goal after a beautiful cross from the left.
And just two minutes later the home fans went celebrating wildly as Luthuli struck a sweet volley from the edge of the area that beat De Jong all ends up.
The hosts had numerous chances to put the game to bed but did not take them and had to settle for victory by a one goal margin.
The victory sees Celtic move up to ninth on the PSL table after collecting seven points from their first five games.
Stellenbosch are still looking for their victory in what is their maiden season in South Africa’s top-flight and sit second from bottom with two points.
For more news your way, download The Citizen’s app for iOS and Android.
Download our app and read this and other great stories on the move. Available for Android and iOS.