/msg is the Internet Relay Chat (IRC) command to send a private message to another user, rather than chatting in a channel. Its syntax is as follows.
/msg <nick> <message>
The underlying RFC1459 protocol message is
PRIVMSG <nick> :<message>
The exact way this is displayed at the other end depends on the IRC client the recipient is using. On the most widespread client, mIRC, the action can be decided using a configuration setting. The most common is for it to open a "Query Window". This is a new window, into which the message will appear. Stuff which the recipient types is then sent back via /msg to the original sender - essentially the window is a private chat window between the two users.
On most IRC server software, communication via /msg is private and can't be snooped on by an IRCop or another user. That said, there's no reason an IRC server couldn't be written that does allow this.