When may the thief Circle? When an ally has dealt damage to the thief's target earlier in the round. Always double (Circle) or triple (Backstab) base die/dice damage first, before adding damage modifiers for Dex, item bonuses, etc.
A thief's Circle damage applies to any target, even non-humanoids (think dragons) or non-living humanoids (undead/golems). Thieves do not get triple damage against such foes, however. Triple damage applies only against vulnerable humanoids.
Sneak/ambush damage for triple the rolled value can only happen if the thief is unencumbered and wearing light or no armor. Circle damage, however, is possible while lightly encumbered and even if the Thief is wearing medium (though not heavy) armor. Backstab requires total surprise. The foe cannot know/suspect that an enemy is able to attack from behind. Only living creatures are susceptible to backstab triple damage. Others may only be susceptible to double damage from Circling.
In group combat situations, it may be possible for thief to make an initial backstab, but in all likelihood he will only be able to Circle for the remainder of the combat. Given the requirement for Circling, a thief cannot Circle when he is alone without allies.