Byron Harrison's second half goal earned all three-points for the Robins against Bury in Sky Bet League Two.
Two first half goals from Clive Platt and Ashley Vincent meant the sides went into the break level in an otherwise uneventful first half.
But when Harrison replaced Jamie Cureton in the second half and immediately came close to taking the lead, the match became a much more entertaining affair.
On that occasion, Harrison forced an excellent save from Brian Jensen in the Bury goal, but the keeper was helpless to prevent the winner, stabbed goalward by the substitute after Jermaine McGlashan got the better of Jim McNulty to supply the striker.
Three points saw the Robins climb back up to 13th in the league and send their fans off with a broad smile.
Yates had made three change following the Robins' 4-1 defeat at the hands of Chesterfied.
Troy Brown rejoined the starting 11 after suffering from a hamstring injury. The defender replaced Mitch Brundle whilst Sido Jombati and Jame Cureton were picked ahead of Craig Braham-Barrett and Byron Harrison.
The visitor's made only one change to the side that overcame Mansfield in midweek as Tommy Miller replaced Chris Sedgewick. That game also ended with a 4-1 scoreline as Bury extended their unbeaten run to six matches.
The first half was a close affair that was marginally edged by the visitors.
Both sides demonstrated fine, passing football that would have entertained the fans much more if either side were more clinical in attack.
Both goals were sloppy. Just after quarter of an hour, Daniel Nardiello danced past Steve Elliott on the edge of the box. The wide man darted to the byline and his ball across the face of goal was tapped in with ease by Clive Platt.
The visitor's were making light work of attacking through the right channel and Robins' fans feared an onslaught after witnessing goals in quick succession against Chesterfield in midweek.
But it was Cheltenham who found the net fastest.
Just two minutes later, Ashley Vincent latched onto a through ball from deep and slotted underneath Brian Jensen in Bury goal.
Both sides lacked the cutting edge in the final third required to find another before the break, Matt Richards' cross come shot came closest to taking the lead for Yates' men but Troy Brown's glancing header was not enough to divert the cross away from Jensen's clutches.
Jake Carroll was denied a goal of the season contender and he turned and shot on the volley. The powerful effort met the head of Richards and glanced over the bar for a corner.
Bury made an early change with 10 minutes left of the first half as Chris Sedgewick replaced Craig Jones. Jim McNulty took over the captaincy after Tommy Miller politely refused to take the armband.
Half Time: Cheltenham Town 1-1 Bury
Jermaine McGlashan emerged from the dressing room rejuvenated as he so often does and was quickly running at Bury's backline.
But the winger's bright start for the Robins was shortlived.
Danny Mayor's low cross found Nardiello but the forward was unable to take the pace off the cross and shot both high and wide.
Mayor was next to try his luck in front of goal as Scott Brown was called into a strong save at his near post.
Robin's supporters on the terraces had their hearth in their mouths as Troy Brown scuffed a sitter.
Deering's shot from range stung the palms of Jensen and the ball was spilt in the six yard box. The offside Troy Brown was unable to put the ball anywhere other than straight at Jensen but any goal would not have stood thanks to the linesman's flag.
The Robin's were unable to sustain their pressure in attack and the visitor's were again on the attack through Nardiello as his shot got the better of Troy and Scott Brown before dropping wide of the far post.
Byron Harrison replaced Cureton just after the hour mark and tested the Bury keeper immediately. Unmarked in the box, the strikers fine header looked goal bound but for an excellent save from Jensen.
The substitue found the back of the net moments later.
McGlashan got the better of McNulty, attacking on the right wing and did extremely well to get to the byline. His square ball into six-yard box was stabbed home by substitute Harrison for the winner.
Jason Taylor replaced Gornell with just 3 minutes remaining, with the aim of tightening up the midfield. It wasn't required as the Robins continued to attack.
Kotwica's selfless link up play with McGlashan on the edge of the box creates space for Jermaine to shoot but his effort is deflected behind for a corner and Byron Harrison forced a reaction save from Jensen to deny the striker a second half brace.
Full Time: Cheltenham Town 2-1 Bury
Cheltenham Town FC (4-2-3-1): S. Brown (GK); Ihiewkwe, T. Brown, Elliott (C), Jombati; Deering, Richards; McGlashan, Gornell (Taylor '87), Vincent (Kotwica '81); Cureton (Harrison '63).
Subs not used: Roberts (GK), Brundle, Braham-Barrett, Williams.
Bury FC (4-4-2): Jensen; Vaseli, Mills, McNulty; Jones (C) (Sedgewick '37), Miller, (Young '79), Tutte, Carrol; Platt (Forrester '69), Mayor; Nardiello.
Subs not used: Hinds, Procter, Young, Burke, Charles-Cook (GK).
Referee: Mr. Andy Woolmer
Attendance: 2,737 (331 Bury Supporters)