There are several ways you can hold a knife. Here you will see some of the advantages and disadvantages in battle situations (I am assuming you use a single-edged blade)
1: Tip forward, edge down
This is how you would hold a bread-knife when you are cutting bread. Or basically any way you would hold a knife.
Advantages: This is how most knives are made to be held, and will probably give you the most confidence. It is a good way for stabbing upwards (to the stomach or throat) or slashing downwards (to the face or arms)
Disadvantages: None in particular
2: Tip forward, edge up
Same as 1, but with the knife turned 180 degrees, so the edge is upwards
Advantages: This is a bit of a surprise hold, and someone seeing you hold your knife like this, will most likely think you are good at what you are about to do. Even if you are not, this is a good thing. It can also be menacing if you can get anywhere near their crotch, because holding your knife this way signifies you are quite happy to cut their balls off
Disadvantages: You remove pretty much any possibilities for slashing jabs, and chances are your knife is not meant to be held this way, and might feel awkward.
3: Tip backward, edge away from your arm
This is the same as 1, except you hold the knife the other way.
Advantages: This stance has a few advantages. For one thing, you can hide your knife behind your forearm, and you can do downward stabbing motions. In these motions, you can put your entire body weight behind your blade, meaning it is quite lethal. You also have the option of making slashing moves across face or neck.
Disadvantages: This is a bad way to hold your knife if you are facing anybody who knows a martial art (this is the first defence against weapon you learn in most martial arts). Also, you will hardly look like a pro.
4: Tip backward, edge toward your arm
This is a strange way to hold your knife, but can be functionable in some situations, especially if the person you are fighting is wearing short sleeve shirts.
Advantages: In addition to the mentioned stabbing motion, you can catch limbs between your forearm and knife, cutting them. Especially effective against people in short-sleeve shirts and throats / necks.
3suns notes that US military trains their troops to use stance 4, tip backward, edge toward arm. The soldier can punch his enemy, follow through past their body, and rake the knife backwards.
Roninspoon, on the other hand, does not agree with 3suns:
I've heard the same thing as 3suns, but I never heard it until I had sperated from military service. It's not really a very good guard or attack stance and would probablly lead to more self inflicted damage than anything else. In retreat, you'd have to strike to the outside of your grip. You'd either have to keep the blade tucked against your arm and present an awkward striking platform that would likely result in a broken wrist, or strike as normal. If you do that though, the back of the blade interferes with the strike, pushing the target away, out of reach of the retreat attack. I think this sort of grip would only work with a double edged blade and even then I have my doubts as to its effectivness as it places your attack well withing the target's defense. Reach is everything in an edge fight after all.
I must honestly admit that I am more inclined to agree with Roninspoon in this case.
Disadvantages: Very awkward way of holding your knife and you might end up cutting yourself
Scenario 1: Both have a knife
In this case, it is time to wonder how the hell you got into this mess. What did you carry a knife around for in the first place? And why did you pull it out when you were in danger?
Oh well. Your first option when faced someone who has a knife is, of course, to run like hell. Probably your safest option, as long as your opponent doesn't have any Hibbens down his boots.
If this is not an option, then the fight begins. I would suggest holding your knife in either position 3 or 1 above, preferably the latter.
Now for the big secret in knife fights:
DO NOT USE YOUR KNIFE.
Yes, you heard me. Do not use it. Your opponent will be so fixed on your knife, that you will easily be able to beat him up with the other hand. Therefore, if you are right handed, hold your knife in your left hand, or vice versa. Then, move your knife away from your body, and watch your opponent follow its tip with his eyes. Now hit him hard in the face with your free hand.
In 99% of the cases (at least in the three knife fights I have been stupid enough to end up in), the fight will be over. Make sure you hit very hard, and assume that they were out to kill you (the knife, remember??). Then take their knife. If they haven't dropped the knife yet, kick them in the knee or in the balls. Then take the knife.
Stroll off casually.
Scenario 2: You have a knife
Ah great. The same applies as for scenario 1, except you have no knife to worry about. Just punch them - hard - and run.
Scenario 3: They have a knife
You are fucked.
no, seriously. You are.
In this scenario, you want to talk soothingly, give them whatever the fuck they want, and if you have a chance, run like hell. And I do mean that.
The best thing you can do, is if they ask you for your wallet, throw it to them, but make sure they can't catch it. Usually you'll be able to run away.
If it comes down to that you have to fight, the trick is this:
Be focused on the knife, but not overly. Make sure they come nowhere near your throat, face or guts with it, but do expect to leave the fight with a few slashes in your arms / hands. Go for their eyes with your fingers, or hit them in the throat. Kick them in all soft spots (groin, particularily) and aim for the nose with your punches.
Then run.
In all scenarios, you want to 1) run away as fast as you can, and 2) call the police to get these maniacs arrested. If you were the maniac with the knife, you should seriously consider getting a life.