The loan capture from Norwich was sent on by boss Darren Ferguson at the break and responded with the decisive strike in the 63rd minute.
Mumba burst forward, swapped passes with Sammie Szmodics and classily fired into the bottom corner to ensure the Championship strugglers eased past their League Two visitors.
Szmodics had earlier give Peterborough the lead, only for Paul Coutts to level with a penalty after half-an-hour.
Posh were quickest to shake off the cobwebs in their first outing since December 18 as they dominated the early stages against a Rovers side appearing after an even longer lay-off stretching back almost a month.
The effects of Covid-19 were still being felt by Rovers, with boss Joey Barton absent after testing positive, leaving coaches Andy Mangan and Kevin Bond to take charge.
The combination of the hosts’ high-tempo passing and the visitors’ dangerously high defensive line made a breakthrough seem inevitable.
It duly arrived after 17 minutes as Szmodics sprung the offside trap to chase an Oliver Norburn pass and tap into an empty net from close range after he and Rovers goalkeeper James Belshaw had failed to make contact initially.
But Rovers are no strangers to FA Cup comebacks, having recovered from a seemingly impossible position in their first-round replay triumph against Oxford before again hitting back to beat Sutton at the next stage.
And they responded well in this tie, with a last-ditch tackle from Nathan Thompson denying Sam Nicholson an excellent shooting opportunity barely a minute after the Posh opener.
Rovers were not behind for long thanks to a coolly-dispatched penalty which provided captain Coutts with his first goal for the club.
The Scottish midfielder, a promotion-winner with Posh much earlier in his career, sent Dai Cornell the wrong way from the spot after the keeper had impeded Antony Evans when dashing from his line.
That surprise leveller knocked Posh out of their stride, but they regained control of the contest in the second half when creating a stream of chances and forging back ahead.
A brilliant Trevor Clarke block denied Siriki Dembele before Belshaw kept out Jorge Grant, but the pressure finally told courtesy of Mumba just after the hour.
Posh should then have killed the tie off but Szmodics could not convert any of three glorious chances, while Grant saw a strong penalty claim waved away.
And they had to survive some uncomfortable moments late on with Cornell denying Rovers substitute Brett Pitman on three occasions to ensure Posh advanced to the fourth round.
The biggest shock was caused by Posh's county neighbours, as Joe Ironside gatecrashed Kieran Trippier’s party when League One Cambridge dumped Premier League Newcastle out of the FA Cup at St James’ Park.
The England full-back was handed a Magpies debut in the third-round tie a day after completing his move from Atletico Madrid, but celebrations over his arrival turned sour as Cambridge heaped further misery on Eddie Howe’s relegation-threatened side.