Christian Eriksen’s deflected strike gave Spurs a 53rd-minute lead at the San Siro, and the visitors looked comfortable for much of the second half.
But Argentinian striker Icardi hammered home a magnificent volley in the 86th minute to draw Inter level, and Matias Vecino took advantage of some slack Tottenham defending at a corner to nod in a dramatic injury-time winner.
“This is exactly what the fans want to see,” said Inter coach Luciano Spalletti.
“Icardi scored a magnificent goal. The team had a great game and really went for it in the end.”
The win leaves Inter second in the early Group B table, behind Barcelona on goal difference after the Catalan giants thrashed PSV Eindhoven 4-0 with a Lionel Messi hat-trick.
The late drama was greeted with deafening applause from the 66,000 crowd in the San Siro who minutes earlier had feared that Inter’s disastrous start to the Serie A season would carry over into Europe.
“We came into this game with quite a few problems,” continued Spalletti, whose side are struggling in Serie A.
“I think that winning in the last minute after turning around the result gives you a lot of enthusiasm and self-belief.
“It means that we can believe in our potential for the future. The reaction is a sign we are on the right path.”
Tottenham coach Mauricio Pochettino lamented his side’s failure to kill off a game.
Star striker Harry Kane missed an early opportunity to add to his Champions League tally after being served up a clever chipped cross from Eriksen on 38 minutes.
“If he had scored maybe we are talking about a different game,” said Pochettino. “We are suffering a little bit but that’s not an excuse.”
Kane, the top scorer at the World Cup with six goals, arrived in Italy having scored nine goals in ten Champions League games.
But with only Samir Handanovic to beat the England striker rounded the Inter goalkeeper before losing his way as the ball rolled behind for a goal kick.
– Icardi’s first Champions League goal –
After Eriksen struck with a deflected shot that looped over Handanovic, the Londoners looked in control.
But the hosts turned things around with Icardi scoring his first Champions League goal in his first appearance in the competition, to add to his 110 goals in Serie A.
Inter Milan, who last won the trophy in 2010 under Jose Mourinho, were returning to Europe’s elite event for the first time since 2011-12, with Tottenham eliminated last season in the knockout rounds by Juventus.
Both teams are faltering in the league with Inter Milan on four points from four games and Tottenham sixth after suffering consecutive defeats.
Pochettino made five changes to the Tottenham side that lost to Liverpool at the weekend with Kieran Trippier, Danny Rose, Toby Alderweireld, Harry Winks and Lucas Moura all dropping out.
“It’s easy to talk about the players who aren’t here,” snapped Pochettino of his tactical choices.
“Against Watford and Liverpool they were on the pitch.
“That is football. After 46 years loving this game, I understand that some tough periods can arrive.
“After the Manchester United win (in August) I said: ‘be careful, be careful the tough period can arrive’ and here it is.
“The important thing is to work harder and to be strong. We have the quality but for different reasons we’re a little late in our preparation.
“Today I started to see good signals that the team is coming back again.
“We showed great personality, playing in the San Siro is a very difficult place and we showed character and personality.”
Download our app and read this and other great stories on the move. Available for Android and iOS.