The expression "house of cards" is also used to describe a fragile or tenuous organization, scheme, or situation. The idea being that like house of cards, any disturbance will cause the whole thing to collapse.