Khulekani Shezi scored the only goal of the game with a cross-come-shot in the 52nd minute that beat Brandon Petersen in the Amakhosi goal.
The Soweto side had missed the last two editions of the MTN8, with their last appearance coming in 2023, when they lost to Mamelodi Sundowns in the semi-finals.
Under former co-coaches Khalil Ben Youssef and Cedric Kaze, the club finished in third place in the Betway Premiership standings last term, while, under new head coach Fernando Da Cruz, Chiefs started their league campaign with a 3-1 win over Kruger United.
Arrows last appeared in the MTN8 in the 2020/21 season, when they also lost to Sundowns in the semi-finals.
They finished in sixth place under Manqoba Mngqithi last term and drew 0-0 with Chippa United in their campaign opener last week, with new head coach Pitso Dladla at the helm.
Chiefs made three changes from their opener, with Nkosingiphile Ngcobo, Wandile Duba and Adolf Mtasingwa Bitegeko all starting, while Arrows welcomed Shezi back from suspension and Junior Dion back from illness.
The home side were more adventurous early on, with Reeve Frosler creating two half-chances around the eight-minute mark after some strong running down the left flank.
Arrows were starting to get into the game with some of their own possession, and Philani Khumalo nearly caught out Brandon Petersen from a whipped-in cross that was heading for the top corner, but the Amakhosi goalkeeper got a strong hand to the ball.
Chiefs, though, found an extra gear in the 14th minute with some superb passing that opened up their opponents' defence, only for the final pass for Luke Baartman to be cut out by Arrows stopper Edward Maova.
Ayabulela Maxwele's back pass nearly caused problems for Maova, with Duba lurking, but the keeper was quickest to the loose ball.
Arrows, though, were certainly starting to find their feet, with Dlamini the creative spark for Dladla's side, but his give-and-go with a lively Ronaldo Van Neel in the 21st minute was just overhit as they entered the Chiefs' 18-yard area.
Mfundo Vilakazi's cross in the 26th minute was calmly cleared by Thulani Mabaso for Chiefs, while Lebohang Maboe had Maova scrambling with a shot from range just a minute later that flew above the crossbar.
Maboe then fired well over the target after the ball dropped to him from the head of Duba, but Ngcobo was left holding his hands on his head after blasting over from close range after being set up by Baartman.
Arrows refused to sit back and were soon up the other end of the pitch, with Khumalo forcing a strong save from Petersen.
Vilakazi was offside as he fired into the side-netting from a really positive spell of attacking play from the home team, while Petersen had to be off his line quickly to cut out a pass from Maxwele, with Van Neel waiting.
Vilakazi then placed a curled effort just past a post with five first-half minutes remaining, with Chiefs seemingly getting closer and closer to breaking the deadlock, while the opening period ended with Ngcobo missing a glorious chance when one-on-one with Maova from inside the six-yard area, after a superb pass from Duba.
Arrows were lively from the restart, with Van Neel trying to make his mark down the left flank, but his low cross in the 50th minute was easily cut out.
But with Chiefs still getting going after a strong first half, they were soon a goal down, as in the 52nd minute a cross from Shezi on the right-hand side was whipped in behind Petersen and ended up in the back of the net.
Maova got both hands to a strong Maboe shot ahead of a double substitution for Chiefs, as Langelihle Phili and Etiosa Ighodaro came on for Duba and Ngcobo on the hour mark, with Da Cruz looking for a way back into the tie.
Maova made light work of another strong shot from a central area from Maboe, as Phili fired wide of the target after a dummy from Baartman, when he was perhaps better placed to shoot.
Phili was closer with an effort he curled just past a concerned Maova a few minutes later, after a better spell of attacking play from Chiefs.
Vilakazi was just about to get onto a superb pass from Maboe that may well have finally opened up the Arrows defence, but the player pulled up with an apparent injury, only to play on after the chance was lost.
Arrows brought on Jerome Karelse and Thokozani Khumalo for Dion and Philani Khumalo, while Chiefs introduced Mduduzi Shabalala for Vilakazi for the final 15 minutes of the match.
Van Neel fired over from the edge of the area as the ball dropped to him from a free-kick, while Maova got down to cut out a Thabiso Monyane cross from the right.
Time was running out for Chiefs to find a response, with Arrows combining some excellent defending with composed play when finding pockets of possession.
Arrows were nearly two goals up in the final minutes as Dion poked the ball goalward, but his effort trailed narrowly wide.
Dillon Solomons came on for Baartman and Siphesihle Ndlovu was introduced for Mtasingwa for the final few minutes, but the Chiefs' changes failed to have the desired effect as Arrows ultimately ran out deserved winners.
