Hemphill was born in Fort Worth, Texas, and attended I.M. Terrell High School (as did Ornette Coleman). He studied the clarinet with John Carter, another I.M. Terrell alumnus, before learning saxophone. Gerry Mulligan was an early influence. Hemphill joined the United States Army in 1964, and served for several years, and later performed with Ike Turner for a brief period. In 1968, Hemphill moved to St. Louis, Missouri, and co-founded the Black Artists' Group (BAG), a multidisciplinary arts collective that brought him into contact with artists such as saxophonists Oliver Lake and Hamiet Bluiett, trumpeters Baikida Carroll and Floyd LeFlore, and writer/director Malinke Robert Elliott.
Hemphill moved to New York City in the mid-1970s, and was active in the then-thriving free jazz community. He gave saxophone lessons to a number of musicians, including David Sanborn and Tim Berne. Hemphill was probably best known as the founder of the World Saxophone Quartet, a group he formed in 1976, after collaborating with Anthony Braxton in several saxophone-only ensembles. Hemphill left the World Saxophone Quartet in the early 1990s, and formed a saxophone quintet.
Hemphill recorded over twenty albums as a leader, about ten records with the World Saxophone Quartet and recorded or performed with Björk, Bill Frisell, Anthony Braxton and others. Late in his life, ill-health (including diabetes and heart surgery) forced Hemphill to stop playing saxophone, but he continued writing music until his death in New York City. His saxophone sextet, led by Marty Ehrlich, also released several albums of Hemphill's music, but without Hemphill playing. The most recent is entitled The Hard Blues, recorded live in Lisbon after Hemphill's death.