With the match seemingly destined to end in a stalemate, Radebe came off the bench to deliver the decisive blow, firing home a superb half-volley deep into added time and sending the home supporters into celebration.
The result puts Milford at the summit of the promotion/relegation playoff standings with six points from two matches. Magesi FC slip to second place on three points, while Cape Town City remain without a point after their opening two fixtures.
Knowing victory would strengthen their promotion ambitions, Milford showed plenty of intent despite Magesi enjoying more possession during the early stages.
The hosts carved out the first notable opportunity in the 13th minute when Siphosethu Ndlabi led a swift counter-attack. However, with sight of goal, the attacker elected to square the ball rather than shoot, allowing Magesi defenders to recover and clear.
Magesi responded positively and threatened through Diteboho Mofokeng, who fired over from a promising position before Mcedi Vandala also failed to hit the target after connecting with a dangerous cross from John Mokone.
As the half wore on, Milford began to find their rhythm. Sifiso Mdluli came agonisingly close to opening the scoring when his teasing delivery drifted towards goal and crashed against the upright.
The visitors then squandered the clearest chance of the first half. Vandala raced through on goal moments before the interval, but Milford goalkeeper Siphamandla Hleza stood tall to produce an excellent save and preserve parity.
Both sides emerged from the break with greater urgency, but clear-cut opportunities remained scarce. Magesi enjoyed spells of pressure while Milford continued to pose a threat on the counter-attack.
A moment of brilliance nearly broke the deadlock in the 77th minute when Bulelani Sigomoshe curled a dangerous set-piece towards goal, only for Elvis Chipezeze to make a superb save.
Just when it appeared the teams would settle for a point each, Milford found one final opportunity. Radebe reacted sharply inside the area and struck a sweet half-volley beyond Chipezeze, sealing a priceless victory and strengthening Milford's bid to reach South African football's top flight.
