This makes sense from a psychological perspective too, as most of Jesus' stuff does.

Swearing vows is an overcompensation--it allows us to metaphorically place a thing between us and our words, thereby removing us by one step from our promises. It's almost like handing off the responsibility, as if when you say, "I swear by all that's holy", you're somehow foisting the responsibility of the truth of your statement off onto the holy things.

Jesus tells us it's plain disrespectful for us to swear by God or God's emanations. This makes sense two ways: It may devalue God or God's creations to use them in this way. It also definitely devalues one of God's creations in particular--Us. When we use other things to buffer our statements and give them artificial power, we are admitting our lack of faith in our own words. According to Christ, if we had but the faith of a mustard seed, we could do everything he does. Not having enough faith to believe your own words isn't the way to get there, apparently.

Jesus knew, I bet, that it takes great courage to simply say "yes" or "no" and leave it at that. It requires that we honor ourselves enough to understand that our word is holy in its own way, because we are (remember?) in the image of God. But of course, if we had even that much faith to begin with, apparently we could move mountains!