Signed out of high school by the New York Mets on June 12, 1964, McGraw became a part of their bullpen a year later. The Mets tried him as a starting pitcher, but he only managed a 2-12 record in 16 starts over two years. After spending all of 1968 in the minor leagues, he became a full-time reliever in 1969. Relying on a good screwball, he racked up twelve saves for the Miracle Mets as they went on to win the World Series, but he did not himself pitch in the Fall Classic.
He became one of the more successful closers in baseball during the early 1970s, placing second in the National League in saves in 1972 and 1973. During the 1973 season, he coined a popular rallying cry for the Mets, "Ya Gotta Believe!" That year, the Mets won the National League East with only 82 wins, but managed to make the World Series, losing to the Oakland Athletics in seven games in a series many Mets fans felt the team should have won!
As a Phillie, he continued his role as a reliable relief pitcher. In 1980, he finished fifth in the NL Cy Young Award voting, compiling 20 saves and a 1.46 ERA and helping the Phillies win the NL East. In the playoffs, he appeared in all five games of the National League Championship Series, saving two of them. His finest efforts came in the World Series, striking out ten batters in 7 2/3 innings. He saved the final game by striking out Willie Wilson, clinching the Phillies' first World Series championship.
He spent the next four seasons as a set-up man rather than a closer, and retired after the 1984 season. He compiled 180 saves in his career, tied for eighth-best in Major League history at the time.
On March 12, 2003, McGraw was working as a spring training instructor for the Phillies when he was hospitalized with a brain tumor. Surgery done to remove it revealed that he had cancer. Given three weeks to live by doctors, he managed to survive nine months. During this time, he attended the closing ceremonies of Veterans Stadium, where he recreated the final out of the Phillies' World Series triumph.
At the time of his death, McGraw was ranked:
- 24th on the all-time major league list in games pitched (824)
- 22nd on the all-time major league list in games finished (541)
- 1st on the all-time Phillies list in games finished (313)
- 3rd on the all-time Phillies list in games pitched (463)
- 4th on the all-time Phillies list in saves (94)
- 8th on the all-time Phillies list in least hits per nine innings (7.89)
- 4th on the all-time Mets list in games saved (86)
- 4th on the all-time Mets list in games finished (228)
- 5th on the all-time Mets list in most games pitched (361)
- 7th on the all-time Mets list in least hits per nine innings (7.78)
- 10th on the all-time Mets list in most batters struck out per nine innings (7.02)