The Boulder office proved more convenient than Nyx's previous digs in downtown Denver: not only was there free parking, initially Darlene Cypser lived only a few blocks away. So, for those times when the machines needed kicking, Darlene usually ended up being the one to give them the boot, leading to many phone calls between Nyx's Boulder office and Trygve Lode's Denver home to go over some of the finer points of kicking misbehaving hardware. In July, 1999, Darlene moved to Denver, which made the trip a little longer, but at least the parking was still free.

In late August, 1999, Arachne (Nyx's webserver) crashed and was not coming back up. Darlene headed over to Nyx's HQ, hoping to bring Arachne back to life. Meanwhile, one of the remote admins was finishing up a kernel upgrade on Iris (Nyx's news server) and by the time Darlene had arrived, he had tried to reboot it...and it hadn't come back up.

Neither Arachne nor Iris would respond to the usual prodding, but while Darlene was trying to get them to come back to life, Nyx's main server, Nyx10, *also* crashed, and crashed hard. Since Nyx's only other login machine mounted most of its drives off Nyx10, and Nyx10 was also handling incoming email, that left very little still running up in Boulder.

Trygve was at a housewarming party at the time and ended up spending the party in the host's basement, trying to talk Darlene through getting Nyx's machines back up and running again...without success.

Darlene packed up all of Nyx's computers and drive arrays into her car, drove down to the housewarming party, where they transferred the machines of Nyx into Trygve's jeep so he could take them home and set them up there. And so, for the first time, Nyx was moved into Trygve's house, and Darlene got to enjoy the rest of the housewarming party.

Nyx in its Boulder location

Nyx in Trygve's Living Room

Simply restoring all of the systems and bringing them back to Boulder probably would have taken about twelve hours. But with everything offline already, it seemed an appropriate time to go through the systems and track down a number of problems and potential problems that had been on the "to fix" list for too long, and to replace every single hard drive involved. In many cases this involved consolidating multiple drives into one, since Nyx was still using several original Quantum Prodrive 105Meg (not Gig, *Meg*) hard drives that had been spinning continuously for nearly ten years.

One way or another, it was possible to recover or restore all the user data and programs and all of the machines themselves turned out to be fine except for Arachne--a Sun Sparcserver 1+ that had been upgraded to a Sparc 5 using an Axil drop-in motherboard replacement. Arachne's new motherboard proved flaky and would occasionally generate memory errors, no matter how many times different memory and different memory configurations were tried, but we were able to get a real Sparc 5 as a replacement.

Four days after all of Nyx's machines were hauled down from Boulder, they were all hauled back up again. Everything came up as planned and Nyx was live once more.

Nyx Moves Again

Nyx under tarps during construction

Nyx's lease at the Boulder location was expiring at the end of January, 2000, and the landlord let us know that they planned to remodel the building, so it would be best for all concerned if Nyx found a new home. Finding another location was a challenge--there weren't a lot of inexpensive locations that already had T1 service and most places that looked affordable but didn't have T1s installed had no available pairs and possibly a multi-year wait before they could get service (if they could get it at all).

But the phone company had just laid down new fiber lines down the highway next to Trygve's house. According to US West, it would be possible to get T1 service installed there in November, 1999. The timing sounded perfect, and it was decided to move Nyx into Trygve's basement. Not only would it cut down on the rent, this way Nyx would be at the same location as all the spare parts and tools and kicking the hardware would be as convenient as a trip down a flight of stairs.

Unfortunately, things didn't entirely work out as planned. November came and went and, each week, US West assured us that the T1s would be installed within two weeks. Then December came and went in a similar fashion...which posed a problem because the lease was up at Nyx's Boulder office and the construction crews were ready to begin.

The landlord was kind enough to let Nyx stay on--since, US West promised, it was only for a week or two--though the construction crew did have to start tearing out the walls and ceiling. The worst part for Nyx was that they kept pulling out the T1 lines, since they figured that nobody was in the room so "obviously" nobody was using the phone lines. Putting up "don't unplug the phone lines!" signs had no effect, so eventually the cords were duct-taped to the plugs. This didn't always stop them from unplugging the lines, but it cut it down a lot.

And so, January came and went, with US West promising all the while that the T1 lines would be installed within a few days. February came and went the same way, except that the remodelling crew was cutting open the exterior walls of the building Nyx was in to install new windows. Nyx kept going through it all under the cover of tarps to keep the dust and debris out of the equipment.

Just before the end of March, US West finally activated the T1 lines. We called out for volunteers and on a snowy March 31st, a team of Nyxers and the Asgard Entertainment stunt team decended upon Nyx's Boulder HQ. At 10:03 AM, MST, Nyx was turned off.

Nyx in Darlene's Car

Nyx moves to Trygve's Basement

Despite the weather and the difficulties involved in dismantling an ISP, Nyx was loaded into cars and arrived at Trygve's house, southwest of Denver, at 12:20PM, not much more than two hours later. At 2:17PM, Nyx was back up and on the net. The dialup lines didn't work--those were being handled through ICG who, we later found out, was routing all our calls to an entirely different call center.

That also took a few weeks to resolve, so while ICG was trying to locate its own call centers, Nyx took over Trygve's fax line so there would be at least one dialup line for the local users.

Nyx Today

In the four years since moving to Trygve's basement, the changes to Nyx have mostly been upgrades and additions to Nyx's hardware. In 2000, the shelving unit full of individual modems were replaced with Boca 33.6 rack modem units donated by Tommy Bowen, in September, Nyx, Arachne, and Nyx10 were all replaced with newer, faster versions with more memory.

In 2001, the Paradyne Channel bank, the Livingston Portmaster IIe--which had developed a few bad ports and finally failed--and the Boca modems were replaced with a US Robotics Total Control Analog modem bank and a Bay Networks Annex 4000. Another machine added to the Nyx stable was a dedicated NFS server to hold all the user files and directories. This way the other machines weren't dependent on one login machine to hold all the user files, which would be inaccessible if that machine went down.

2002 saw even more changes on the hardware front. The router was changed to a Cisco 3620, and the old mailserver (a Sparcserver 1+) was relaced with Anubis, a Pentium-II/400 based server with eight 4.3G drives running hardware RAID-5. The modems were replaced again, this time with a Livingston/Lucent Portmaster 3, upgrading the dialup lines to 56k/v90. Three new login machines (Nyx1, Nyx2, and Nyx3) were added, each one being dual-300 Sun Ultra 2s, and the webserver was upgraded to a dual PIII/850 machine with 2 Gig of memory and hardware RAID. Nyx also added a dedicated backup server with a 80-gig "spool" drive and a pair of DLT libraries, of many improvements that Casper Maarbjerg has set up for Nyx since joining Nyx's team of admins in 2001.

Nyx Login Machines

Nyx Network

By 2004, even the new mailserver was straining under the ever-increasing load caused by junk email, so it was once again replaced, this time with Hermes, a dual 1.1GHz PIII machine with 2 gig of memory. The news server suffered multiple simultaneous hard drive failures (they'd been running continuously for quite a few years) so a new one was built from scratch, in this case a dual PIII Xeon/550 with 1 gig of memory.

...TO BE CONTINUED