Since mars has an atmosphere, and it is composed mostly of a greenhouse gas, why isn't there a significant greenhouse effect to warm its surface?

While Mars' atmosphere is mostly CO2, a greenhouse gas, there isn't enough atmosphere to create an effective insulating layer to hold in the trapped IR radiation.

Explanation:

The remaining gases are a mixture of nitrogen, argon, oxygen and carbon monoxide. CO2 is a potent greenhouse gas, so Mars does have a greenhouse effect. But it's very weak because the Martian atmosphere is so thin -- 100 times less dense than the Earth's atmosphere.