Oooh. I've been there, done that. That really hurts. I had just moved up to admin a few weeks before, and I was training one of the techs to do my previous job of tech supervisor. We were sitting there in my 10x10' cell, (they call it an office; I call it a small prison near the server racks), and I was about to add him to sudo for the mailserver, so he could fix a few things if they happened (damaged mail, add aliases, etc). I ssh into the server, still talking to him about something, and not really paying attention, and type `su`. I'm going a mile a minute and totally miss the 1/4 second delay between typing `su` and typing in my pass - and the root pass is displayed right there.. AAagh. The load on the machine was pretty high, because as is often the case, management fails to listen to the network guys when it comes to spending money. We needed a new box so we could balance the mail load, but they didn't want to spend more money at the time.. so there was sometimes a bit of lag on the system.

jamyn@sys48$ssh jamyn@mail
jamyn@mail's password:
Last login: Mon Apr 4 17:11:32 1999 from sys48
Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 4.0-RELEASE (KERN/QM-BSD4)

jamyn@mail jamyn su
WtfwytysfYchtPassword:
Sorry
jamyn@mail jamyn logout


What a heartbreaker. I heard a slight "heh" behind me, and knew the tech was trying furiously to memorize the password, even as I was slamming Control-D as fast as I could, to logout and clear the screen. It took all of 1/2 a sec for me to realize the mistake and the screen was cleared, but it doesn't matter though.. I couldn't keep that pass anymore of course. "What the fsck were you thinking you stupid fsck, you cant have this" ; "WtfwytysfYcht". I told the employee to go back to the tech room, I had some stuff I needed to do and would meet with him in 15 minutes. I then chose a new root pass for the mailserver, and emailed it to the rest of the NOC staff.

Heh. You're thinking, 'you did what?! you EMAILED a root pass?!' .. well yea, I dont feel so bad about it; you see - all NOC mail is PGP encrypted with a 4096 bit key. Not just some messages -- all messages to NOC. That includes system email's from any box (using PGP wrappers for sendmail), etc etc. The core NOC staff keep the private key, and use it to decrypt the messages; it works out pretty nicely. Ah well, live and learn. =)