2000-11-16: The .to administrators destroy yp.to, for reasons that have not been explained. They try to fix the problem quickly, but it persists for many hours, thanks to BIND's obsolete zone-transfer protocol.
2002-01-20: It happens again. Aargh!
The cr.yp.to web pages were unreachable during these outages. If your browser's error messages misled you into believing that cr.yp.to didn't exist, complain to your browser vendor.
1998-06-03: Power outage at UIC.
1998-10-13: Power outage at UIC.
1998-11-23: Power outage at UIC.
1999-01-07: Power outage at UIC.
1999-07-06: Power outage at UIC. This one lasts for eight hours, which I believe is a UIC record. Rumor says that the technicians don't like hearing the generator alarms so they turned the alarms off. Rumor also says that UIC has owned a backup generator for a while but has never plugged it in.
1999-10-22 ~14:00 GMT: Power outage at UIC. A short one, only two hours.
2000-01-01 04:00 GMT: UIC's network goes down for four hours. This is a PHB problem, not a Y2K problem. As explained by Ahmed Kassem two weeks earlier: ``UIC's campus network has been tested ... we fully expect the network to remain completely operational ... we will disconnect UIC's network from the outside world ...'' That's an interesting definition of ``completely operational.''
2000-01-10 ~02:00 GMT: Scheduled upgrade to OpenBSD 2.6. One monitor explodes in the middle of the operation, but fortunately I have a spare.
2000-01-18 10:16 GMT through 13:00 GMT: One of UIC's routers dies, cutting the department off from the Internet.
2000-01-29 16:10 GMT: Power outage at UIC.
2000-04-13 ~10:30 GMT: Power outage at UIC.
2000-06-30 00:00 GMT: A planned 10Mbps-to-100Mbps upgrade is stymied for a few minutes by a low-quality network cable.
2000-08-23 ~04:00 GMT: OpenBSD crash. Cause undetermined.
2000-10-08 17:45 GMT: Power outage in downtown Chicago. Power is restored five hours later. The computers are revived the next day.
2000-11-05: Network outage for more than 24 hours, thanks to Ameritech, the local phone company.
2000-11-22 11:00 GMT: Network outage for more than 4 hours, thanks to Ameritech.
2001-01-05 19:45 GMT through 21:10 GMT: OpenBSD crash. Cause undetermined.
2001-01-12 00:00 GMT through 00:25 GMT: Scheduled upgrade to OpenBSD 2.8.
2001-03-14 22:30 GMT through 22:40 GMT: Hardware upgrade.
2001-04-25 21:00 GMT through 21:10 GMT: Scheduled electrical upgrade.
2001-05-04 18:35 GMT through 18:40 GMT: Scheduled electrical upgrade.
2001-05-23 21:00 GMT: Scheduled electrical upgrade. Reboot is prevented by hardware failure: the mail disk super block has died. Web service is restored within an hour. Mail service is restored the next day.
2001-07-08 ~00:00 GMT through 17:35 GMT: Power outage at UIC. Power is restored at some point between four hours and seven hours later.
2001-07-30 ~19:30 GMT through ~23:30 GMT: Network outage at UIC. The computer center is more informative than usual: ``There is a hardware problem with the ATM link leaving campus. We do not yet know the exact cause or when our external link will be fully operational.''
2001-08-26 ~04:00 GMT through ~19:00 GMT: OpenBSD crash. Cause undetermined.
2001-09-06 04:00 GMT through 04:10 GMT: Hardware failure.
2001-11-03 15:00 GMT: Fiber cut at UIC. Fixed 50 hours later.
2002.02.26 ~17:30 GMT through ~19:30 GMT: OpenBSD network stack crash. The load was not heavy (about 20 web downloads per second from slashdot, plus a few mail deliveries per second) and presumably would have been handled without trouble by the FreeBSD network stack.
2002.03.09 09:39 GMT through 17:02 GMT: Power outage at UIC. Didn't hit this building, amazingly enough, but did take down our Internet connection. Didn't quite match the previous eight-hour record.
2002.03.24 ~10:00 GMT: Network outage at UIC. For at least the first hour, the computer center claims that everything is just fine.
2002.08.27 through 01:59 GMT next day: Crash. Cause undetermined.
2002.09.11 through 17:57 GMT: Crash. Cause undetermined.
2002.09.18 20:40 GMT through 20:45 GMT: Scheduled outage. Web pages are moving to a FreeBSD machine.
2002.09.27 14:00 GMT through 16:00 GMT: Scheduled mailing-list outage. Mailing lists are moving to a FreeBSD machine.
2003.07.17 22:30 GMT through 2003.07.18 07:30 GMT: Power outage at UIC (under 2 hours), followed by network outage at UIC (over 7 hours).
2003.08.14 23:45 GMT through 2003.08.15 03:30 GMT, with occasional uptime: Supposedly a scheduled UIC network outage to install uninterruptible power supplies for the UIC routers. No explanation of why this takes four hours.
2003.10.11: My DNS cache is actively disabled by the idiots at the UIC computer center because it is ``scanning remote hosts on port 53.''
2003.10.14: Network outage at UIC.
2004.09.14 09:30 GMT through 2004.09.15 17:30 GMT: Massive power outage at UIC. (``A water main break flooded electrical vaults on the east campus.'') The power outage lasted roughly 12 hours; the rest of the delay was caused by a UNIX filesystem that had blocks written in an incorrect order. (This could have been a bug in the disk hardware, such as the idiotic idea of disk-write caching, or it could have been a bug in the filesystem software.)
2004.12.07 19:05 GMT through 19:15 GMT: Power outage at UIC. One of my computers died, extending the outage a bit, despite the UPS; I wonder whether a power spike hit the computer through the network.
2004.12.09 12:05 GMT through 12:15 GMT: Power outage at UIC.
2004.12.26 several hours: Hardware failure. Equipment upgrades are being stymied by UIC's incompetent grant management.
2005.08.14 about an hour: My web server is actively disabled by the idiots at the UIC computer center because my machine is running a ``rogue FTP server.''
2005.10.12 about two hours: My web server is actively disabled by the idiots at the UIC computer center because my machine is ``compromised'' and is ``scanning on port 25.''
2006.03.16 about 16 hours: My mail server is actively disabled by the idiots at the UIC computer center. Could some masochist please hire Ed Zawacki so that I can stop dealing with his incompetence?
2007.01.09 about 12 hours: For unexplained reasons, UIC disables all network traffic except to the computers at the UIC computer center.
2007.03.18 ~18:00 GMT: Scheduled outage to move servers.
2007.03.20 ~23:00 GMT?: My mail server is actively disabled by the idiots at the UIC computer center.
2007.09.29 ~04:00 GMT: Another multiple-hour power outage at UIC. The explanation, which you're not supposed to laugh at, was finally given on 3 October: "There are two ComEd grids that feed the UIC campus. One of the grids went down causing a power glitch for 2.5 minutes. As a result, many building level breakers tripped resulting in the loss of power on a building basis that lasted until the UIC electricians could reset the building equipment." You're also not supposed to laugh at the fact that, several hours after the outage, the computer center still wasn't aware that there had been an outage.
2008.05.05: One of my servers is actively disabled by the idiots at the UIC computer center.
2008.11.08 ~13:30 GMT to ~17:20 GMT: Intentional multiple-hour power outage at UIC, announced on a web page just 16 hours in advance.
I am interested in hearing price quotes from Chicago ISPs for independently providing a wire to the math building with various levels of network service. We're at 851 S. Morgan.
2000-09 through 2000-11: UIC is paying its ISP for only 14Mbps, and is hitting that limit more and more frequently. The packet-loss rate hits 25% at busy times. The UIC computer center refuses to pay for adequate network service. Perhaps the router is being flooded by some easily fixed source of traffic; the UIC computer center won't even bother investigating.
2001-05-19 through 2001-05-21: Several reports of overloads. UIC is paying its ISP for 20Mbps.
2002.11.04 through 2002.11.07: More reports of overloads. The UIC computer center issues a statement: ``Over the past two days the University has experienced degraded network connectivity ... Initially, this was the result of an unsanctioned and unannounced network experiment between researchers at UIC and a Greek research network which flooded our outside connection. Eventually, we managed to stop the traffic flow from bringing down our outside connections. In the meantime, our border router's connection to the rest of the campus had become affected in such a way that it was performing dramatically worse than before. After unsuccessfully attempting to fix the issue using a variety of means, including hardware replacement, the decision was made to completely redo the way this router connects to the University backbone. Only then was the problem completely resolved.''
2002.11.13: Horrible network performance at the edge of the UIC network; no packets lost, but ping times up to 2 seconds. In the evening, periods of complete outages. The UIC computer center issues a statement: ``Over the past 2 days, we have been having enormous difficulties in dealing with a Denial of Service attack that affected our ability to route traffic off of campus. After working with our two primary ISPs, we have finally been able to resolve the routing issue, and traffic is now leaving campus normally.'' Outages continue for a little while after that statement.
2005.11.22: Someone at IP address 196.34.112.44 (in Sentech's network) spends three and a half hours making 140000 connections to the cr.yp.to SMTP server with some impressively incompetent spam software. Slight degradation of service, nothing serious.