Just yesterday i was participating in the elven T4-Zone, Destro managed to get a 2-star keep pretty quickly while the order keep was still at 0-star. But then for the longest time just nothing happened, like at all.
The siege only finally began like 2 hours later, when the order keep was already at 2-stars and after they had a ram of their own destroyed.
Is there some huge downside to a failed siege, that I am not aware of? (Debuffs, Keeps deranking or what have you) Otherwise I don't understand while you atleast just wouldn't try a siege, especially while the enemy keep is still low level, because seems to me, that a failed siege doesn't really cost anything.
Please help me understand this.

Thank you!