The Perils of Popularity, or: How Is a Frozen Website Like a Sick Person?

It is hard to predict, in nearly every pursuit, what will be popular and what will not. Blog posts are no exception. Sometimes a blogger posts something that would seem to generate a lot of interest and it fades without a trace; sometimes you post something that seems like no big deal and, for whatever reason, people care a lot.

The latter case happened here the other day, with a post about libraries. The post was circulated via Reddit, which, if the wind is blowing just right, can be the equivalent of dropping a match in a dry pine forest. Indeed, Reddit spread the post so quickly and well that our website was overwhelmed and, for the better part of 24 hours, was inaccessible, even to us. We couldn’t even get onto the site of our hosting service, 1and1.com, to see what was going on.

But that’s what I learned in retrospect; at the time, we were baffled.

When I was finally able to access the hosting site yesterday, I found a neat example of how illusory data can be. Let me explain:

The first graph I looked at showed the number of daily unique visitors (excluding feed readers) on Monday and Tuesday of this week. It was on Tuesday afternoon that we began to have overload problems. But this graph alone didn’t indicate anything very unusual: there were about 60,000 visitors on Monday and 70,000 on Tuesday.

Next I looked at the hourly traffic for Tuesday. It, too, looked normal, growing in late morning EDT, peaking at about 3 or 4 p.m. EDT, but overall showing a pretty gentle upward slope and then a gentle downward slope. Again, if you looked at this graph alone, you wouldn’t have suspected anything out of the ordinary. Why? I assume it’s because the nature of the overload meant that some people were getting through, but it took a long, long time.

(I would show you these two graphs I refer to above but, sadly, 1and1 is still not providing full service to us, so I can’t; I did, however, download two images that are pasted below.)

But then I looked at the site data for Referrer URL’s — i.e., where people are on the Internet before they land on our site. This data usually only reflects about 10% of our incoming traffic — it doesn’t tell you when people come to your site from their own bookmark, for example — but it’s still helpful. Here the Referrer data for Monday, which was a normal day:

And here is the Referrer URL data for Tuesday:

Hello, Reddit! Once you see this picture, the cause of our overload is pretty obvious; but the first two layers of data I looked at offered no clues as to the nature of the problem — or even the fact that there was a problem. This tiny and insignificant riddle made me think about how hard it is for a doctor to diagnose a problem in a patient, and why I was so impressed with Jerome Groopman‘s book How Doctors Think, which is an exploration of that exact problem.

Apologies to all of you who had a hard time getting onto our site, and apologies also for not posting during our outage. The good news is that we will soon be making a pretty significant change around here, to be announced in a few weeks, that should ensure against this type of problem in the future.


Cliff

What do you think of 1and1? Would you recommend them?

LLP

It's 11:50 am EDT 7/12/97 and I'm still getting errors trying to access your website. When I do get connected, links don't work and comments are being dropped or rejected.

ithaca99

Perhaps Taleb should consider the question of black swans among blog posts.

sygyzy

1and1.com? That's your problem. Let me guess, you guys have Capital One credit cards too?

giromide

1and1.com isn't bad. I enjoy reading their broken English in correspondence. The only thing I find annoying is their handling of spam filtering, but they have made several improvements during the last year.

eerac

It seems like you guys crashed because you have an insanely large image (1.5 MB) loading every time someone opens the libraries post. The exact same thing happened on a site I help run (passiveaggressivenotes.com) a few weeks ago. We had about 40,000 visitors when reddit linked to this post.

If you guys replace the large image in the post with a smaller one, things will be much better. Since the image gets scaled down by the html anyway, having a full size image load is a total waste.

You can also put images on photosharing sites like flickr, and display the image using the flickr url. For most people hosting their own blogs, storing images offsite makes pages load a lot faster.

frankenduf

are you sure it wasn't the large breast posting that caused the backup?

RealityCheque

It shouldn't be so surprising that this post generated traffic. Anything new on the internet that's even remotely related to libraries enjoys ridiculous popularity. Case in point: that inane NY Times Style article on "young" librarians ("A Hipper Crowd of Shushers," June 8) has hung on in the Top 10 E-mailed list all week, and has actually jumped back up to no. 1 more than once.

Then again, what else would you expect from a group of people who sit in front of a computer in an extremely quiet room all day?

RealityCheque

That should be June 8 and close parentheses. There isn't anyhting 8) -worthy in that article...

RN3AOH

Properly tuning your Wordpress installation helps a lot to survive such storms. There's a lot of pages devoted to this around, and it would be a good idea to look them up.

Kent

:-(

hyperzx

LOL thats too funny that story looked so good in my RSS reader and I accidentally removed it and was so pissed I couldn't get on the site all day ;)

aardvarkious

should I take blame or credit?

Cliff

What do you think of 1and1? Would you recommend them?

LLP

It's 11:50 am EDT 7/12/97 and I'm still getting errors trying to access your website. When I do get connected, links don't work and comments are being dropped or rejected.

ithaca99

Perhaps Taleb should consider the question of black swans among blog posts.

sygyzy

1and1.com? That's your problem. Let me guess, you guys have Capital One credit cards too?

giromide

1and1.com isn't bad. I enjoy reading their broken English in correspondence. The only thing I find annoying is their handling of spam filtering, but they have made several improvements during the last year.

eerac

It seems like you guys crashed because you have an insanely large image (1.5 MB) loading every time someone opens the libraries post. The exact same thing happened on a site I help run (passiveaggressivenotes.com) a few weeks ago. We had about 40,000 visitors when reddit linked to this post.

If you guys replace the large image in the post with a smaller one, things will be much better. Since the image gets scaled down by the html anyway, having a full size image load is a total waste.

You can also put images on photosharing sites like flickr, and display the image using the flickr url. For most people hosting their own blogs, storing images offsite makes pages load a lot faster.

frankenduf

are you sure it wasn't the large breast posting that caused the backup?