they received the greatest value from interacting face-to-face. By being in the same room, people could quickly shift from conversation to conversation when a critical phrase was heard, with a very low barrier to asking someone a question or suggesting an idea.

Although the crit-sit seems heavyweight and wasteful, we have no other approaches that can replicate the collaborative interaction of a bunch of people stuck in a room searching for a solution to a common problem. It would be a revolutionary advance for system administration if a tool were developed that could permit the same engagement in remote collaborators as we saw in the crit-sit room.

We next describe the sorts of collaborations we observed among security administrators at a U.S. university.

the “ettercap” incident

When we first met the security administration team for a computer center at a large university, 5 they seemed somewhat paranoid, making such statements as, “I’ll never type my password on a Windows box, because I can’t really tell if it’s secure.” After watching them for two weeks, we realized they had good reason to be cautious. IT systems, as a rule, have no volition and don’t care how they’re configured or whether you apply a patch to them. Security administrators face human antagonists, however, who have been known to get angry when locked out of a system and work extra hard to find new vulnerabilities and do damage to the data of those who locked them out.

The work of these security administrators was centered around monitoring. New attacks came every week or two. Viruses, worms, and malicious intrusions could happen anytime. They had a battery of automatic monitoring software looking for traces of attacks in system logs and network traffic. Automated intrusion-detection systems needed to err on the side of caution, with the sysadmins making the final decision as to whether suspicious activity was really an attack. These sysadmins relied on communications tools to share information and to help them maintain awareness of what was going on in their center, across their campus, and around the world.

The security administration team shared adjacent offices, so back-and-

one of our
motivations
for studying
sysadmins is
the ever-increasing
cost of it
management.
Part of this can
certainly be
attributed to
the fact that
computers get
faster and cheaper
every year, and
people do not.

forth chatter about system activity was common. They joked about taking down the wall to make one big workspace. They also used a universitywide MOO (multiuser domain, object oriented), a textual virtual environment where all the system administrators would hang out, with different “rooms” for different topics. The start of an incident would result in high levels of activity in the security room of the MOO, as security admins from different parts of campus would compare what was happening on their own systems. On a day-to-day basis, the MOO might hold conversations on the latest exploits discovered or theories as to how a virus might be getting into the network. The admins described the MOO’s persistence features as really helpful in allowing them to catch up on everything that was going on when they came back after being away, even for a day. They also used a “whisper” feature of the MOO for point-to-point communication (like traditional IM).

An example of MOO use for quick interchange of security status came when we observed a meeting that focused on hacker tools. The security administrators discussed a package called “ ettercap.” Being unfamiliar with this tool, one of us began searching the Web for information about it using the wireless network. A few minutes later, one of the administrators in the room informed us that a security administrator working remotely had detected this traffic and asked about it on the MOO:

 

Remote: Any idea who was looking for ettercap? The DHCP logs say [observer’s machine name] is a NetBIOS name. Nothing in email logs (like POP from that IP address).

Remote: Seemed more like research. Remote: The SMTP port is open on that host, but it doesn’t respond as SMTP. That could be a hacker defender port. Local: We were showing how [hacker] downloaded ettercap. One of the visitors started searching for it. Remote: Ah, OK. Thanks.

 

In the space of only a few minutes, the sysadmin had detected Web searches for the dangerous ettercap package, identified the name of the machine in question, checked the logs for other activity by that machine, and probed the ports on the machine. He could see