or, "When there's no more room in Hell, the servers will walk the Earth."

Five years ago, Daily Kos was in dire straits. Booming traffic kept bringing the site to its knees. First DKos moved from a shared server with lots of sites to a server with only a few higher traffic sites. That failed to be enough, though, so after flirting with Slash, Markos raised $2500 (surpassing the original goal of $1000) to purchase a dedicated server to run the site and handle the transition to Scoop.

On Thursday, that server was shut down for the last time.

The writing had been on the wall for quite some time. Every once in a while, swish-e processes would eat up all the available memory and CPU and the server couldn't recover, since while one process eating all the memory and CPU would be killed, another would be waiting to take its place. Going into a death spiral, the server would cease to respond and have to be power cycled to recover.

There were other fears with it as well. If anything broke on it, replacement parts would be difficult to come by. Expansion was a non-starter. Finally, it was stuck running FreeBSD 4.11 and could not be upgraded any further because the 88MB root partition simply did not have enough space to permit trying to upgrade across major revisions.

Once, it ran all of Daily Kos. As time went by, its role shrank until it just ran outgoing mail for the site and jotter's search stuff. Despite its age, years of accumulated cruft, and struggles with its workload, it soldiered on for many years.

Plans had been in place to move mail and search to a new server for a while, but such a major move requires planning and thought. On Wednesday, all those plans went out the window. The old server froze up again for the second time in two days, so I went and hard rebooted it since nothing else would bring it back in that situation. I waited and waited for it to come back up, but it never did. Growing a little concerned, I connected to its serial console and saw that it was waiting at a single user prompt - /var was corrupted. I ran fsck to fix the errors as suggested, but it was bad. Very bad. Scores of hard read errors scrolled by while I watched helpessly, the superblock was completely hosed, and the "lost+found" directory was gone. I was also given the impression by the folks at the data center that that server was making a loud racket while I was trying to resuscitate it. I ran it a second time, but all the same errors went passing by while I sat and waited for it to get fixed. It wanted me to fix the filesystem again, but I could tell it was never going to get properly fixed, so I tried rebooting. Miraculously, it came back. Why it managed to come back I have no idea, but it gave jotter and I the time we needed to get the important stuff off of it before it failed for good. It even kept running until I pulled the plug on it once the switch was complete, still working away.

Its service may not be over yet. It's going to be sent to me, and I'll perform surgery on it and try to bring it back to life. If it's only a busted hard drive, then it'll be back on its feet serving Daily Kos again in its retirement, working as a development box in its old age. That $2500 server from five years ago might not be done yet, but even if it is it's had a long career. Take a moment today and pour a forty out on the ground, or sacrifice a bull, for the computer that worked for you just as long as it could.

And the lucky winner is...

lineatus with a picture of a bird! This time, it took just a hair under 5 weeks to add another million comments.

I'm afraid I'm not feeling super creative today, so this diary is sadly lacking in humor. Uh, so carry on.

Warning: This article is not for the faint of heart. It is a technical discussion of webserver optimization and setup, so if you are not already familiar with the subject, most if it will not make sense to you. Rather, it is presented with the hope that those Daily Kos readers who administer webservers, either their own or for work, might find it useful.

Scaling a website or blog is a constant battle. One of the first things people do is set up a two-tiered system on their webservers, with a lighter proxy httpd server sitting in front of the heavy duty backend application httpd. This way, the lighter webserver program can handle the common lighter requests, reserving the backend for the harder dynamic tasks. Serving up things like images, javascript files, CSS files, and other static content with a light weight process without the overhead of a full mod_perl/mod_python/etc. installation can increase performance dramatically.

The two-tiered setup with webservers is common knowledge. As time goes by and your site's traffic starts increasing, however, performance will begin to suffer again and you'll have to find different ways to scale.

A common way to do that is to cache dynamic content for anonymous users. A good rule of thumb is that anonymous browsers make up 90% of your website's traffic, so if you can avoid regenerating those pages constantly for anonymous visitors you can theoretically reduce your website's load to a tenth of what it was. Writing pages out to disk is a good way to cache pages, but it isn't perfect. If you're going to cache more than just the index page and other pages found at a consistent URL, you'll have to find a way to dump the pages out to disk and use some rewrite trickery to properly serve up the cached pages. If you have more than one webserver in a pool, then the cache isn't shared between the machines. Finally, and this was becoming a problem on Daily Kos, you can have performance issues caused by dumping the files to disk by itself when you have multiple webservers all updating their caches simulateously.

I solved this problem by moving from an Apache 1.3 proxy to a hacked up lighttpd proxy using mod_magnet and memcached to cache anonymous pages, sharing the work of caching them between all the webservers, and greatly expanding how much content I could cache. Below the fold is a description of how it works, how to set it up for yourself, caveats, and ideas for future development.

Note: This works for me, but like all things computer, it may not work for you for no clear reason whatsoever. Always make backups and have a disaster recovery plan in place.

As diaried previously, there are a limited number of scholarships to send people who would otherwise be unable to make it to Netroots Nation this summer available. Building on kid oakland's heroic efforts to send bloggers to Yearly Kos last year, the scholarship program has returned.

The deadline for applying has even been extended to tomorrow, so if you'd like to apply you still can. If you are going to apply, though, now would probably be a good time. I'm all for waiting until the last minute to do something, but that minute is almost here.

Don't forget that you can also vote for your favorite nominee at the same site. You can help decide which worthy user gets a trip to Netroots Nation.

If you'd like to help some other way, be it donating money, frequent flyer miles, a NN registration you had purchased but can no longer use, or that can of lima beans that's been lurking in your pantry for years (even bloggers need to eat), Land of Enchantment is coordinating the donation efforts. There's nearly 100 applicants for 9 scholarships, so anything you can do to help will be appreciated. Donations can even be made after the application deadline, so more scholarships will be able to be awarded.

Remember, though: if you're going to apply, you have to do it by tomorrow, June 10th. After that, you'll have to get to Netroots Nation on your own, be it by horse or cow. If you don't want to ride a cow to Austin, apply now. If you don't care that some fellow blogger will be riding a cow to Austin, perhaps your donation will help them ride a better cow.

Earlier today, the 19 millionth comment posted on Daily Kos was posted by SunWolf78, discussing reasons that Hillary Clinton should not be nominated to be the Vice Presidential candidate.

It took from March 27th until May 7th to go from 18 million to 19 million, clearly a slackening pace compared to the 5 weeks it took to get from 17 to 18 million, and one month (almost to the hour) to get from 16 to 17 million. It remains to be seen if this drop is due to the death of Daily Kos being at hand, or because there wasn't a whole lot going on for much of March and April. The minimal traffic we've had lately has been pretty devastating though. It's like a ghost town around here.

CNN aired some footage of Hillary Clinton's campaign headquarters from tonight after getting the news from Indiana. It remains to be seen what the Clinton campaign's actions in the coming days will be after tonight, but as her speech tonight indicated, clinton seems to be experiencing some mood swings. This is one of those times where the video speaks louder than words (as it were).

Scientists announced earlier this week that a new tallest mountain in the world has been discovered, dwarfing the height of the previous record holder, Mt. Everest, by almost 100,000 feet.

Rising majestically from the lowlands of Armonk, NY, the newly discovered Mt. Worsethanhitler soars a full 121,451 feet into the sky, far exceeding the 29,029 ft. of Mt. Everest.

Little is known of the peak besides its great height, but George Seals, one of the scientists who announced the discovery of the mountain, is preparing an expedition to attempt a summiting of the mountain. No one is sure what to expect, but they have hired one of the local tribesmen, a man named "Steve", to help them reach the top. It is believed, however, that they will likely have to tread carefully on the mountain, lest they squish any of its geographic formations. "Yetis, or any other manner of vicious beasts unknown to Western science, may lurk waiting on the mountain's slopes. We must be cautious," Seals said.

When asked why the mountain had never been noticed in previous centuries, Seals replied "perhaps no one looked directly at it. For all we know, there could be even greater mountains we haven't seen yet, sitting in unexplored backyards across America."

Bloggerheim ist ihr Land.

I was battling with zlib last week, so I wasn't on the ball and missed this. In a development that will likely change the course of the primaries and give the Obama campaign a well deserved boost, icebergslim posted Daily Kos' 18 millionth comment back on the 27th. Congratulations.

Apologies for the machine translated German. Please refer to Siegfried, Act I, scene ii for an explanation.

Or, a look at what I've been working on for the site.

It's been three months or so in the making, but today I've finally finished moving the site to its new hardware. Many of the major and minor things that could go wrong, went wrong, but at last everything's squared away and we're up and running on the new machines. Now that that's finished, I can even start rolling out some new features that I've been waiting on until this was all finished.

WARNING: There will be honest, frank discussion of computers and technology issues in this diary. Please, for the love of God, do not post a comment saying that you don't understand what I'm talking about. It will only antagonize the violent beast within.

Let's get started.

Chicken Suit was Beginning.

Congratulations 417els. Amazingly, the 17 millionth comment was neither about a candidate, a picture of someone's cat, nor consisted solely of the word "heh".

In a shocking turn of events, comment #17,000,000 followed quite close on the heels of comment #16,000,000, separated only by a month (described here). It took a month and a half to get from 15 to 16 million comments, and for a while before that it was fairly consistent at one million comments every two months. A while back, I made a handy chart showing the number of comments over time, but I'll need to update it before I post it.

The Germans, after a month has passed since the winter solstice, celebrate a most curious custom. They gather at each others' homes, and at the temples, and the fields, and eat, and drink, and make sacrifices to Mercury. They shout unto the heavens for many hours, and throw swine back and forth until the sun sets.
-- Gaius Cornelius Tacitus, De origine et situ Germanorum

I have the strength of twelve men, for my defensive line is pure.
-- St. Vincent Lombardi

10th Century Icon of St. Vincent Lombardi. The representation of the Chiefs as a Dragon is allegorical.

Most people are aware of the pre-Christian origins of many of our holidays, like Christmas, Easter, and Halloween. Many have forgotten, though, that one of our most important holidays also has its origins deep in the mists of history, long before Jesus of Nazareth began preaching in Judea. I refer, of course, to Super Bowl Sunday.

The Roman historian Tacitus tells us that as part of the Germans' celebrations for their god he identified as Mercury (believed by modern historians to be the same as Wotan in the Germanic pantheon, or Odin from Norse sources), they would toss pigs back and forth while drinking and feasting. For reasons that are obscure, during the Migration Period this grew to become one of their most important religious festivals, eclipsing the ceremonial duels between the sixty five greatest warriors of the land held every spring, and even the Serius Mundi, as Roman authors called it, held every fall.

As Christianity began to spread among the Germans, the Church realized that they would not easily give up their most important holiday. Thus, as they would later do with Christmas and Easter, they adapted Super Bowl Sunday to a Christian theme to speed Germanic acceptance of Christianity.

The saint whose career proved to be the best match for Super Bowl Sunday was Vincentius of Langobardia, commonly known in English as St. Vincent Lombardi. His early life is obscure, but he was probably born ~610 AD in Milan. He first enters history in 663 AD as a general in the armies of Grimoald, King of the Lombards. The Lombards had had many successes in Italy, but were facing an invasion by the Chiefs, a still pagan tribe from the East, led by their king Henricus Strammo. On account of Grimoald's illness, Vincentius was charged with defending Lombardy from Chieftain depredations.

On a January afternoon in 667, the armies met in battle. The first half of the battle, according to contemporary accounts, was fierce, with the Chiefs almost gaining the upper hand. During the second half, though, a series of successes led to Vincentius defeating the Chiefs and forcing their King to accept baptism. Chastened, the Chiefs retreated.

For his role in saving Lombardy and the rest of Italy, Vincentius was honored by the Pope and given a ring. (Vincentius was Catholic, but his religion was tolerated by Grimoald, who, while an Arian, was married to a Catholic.) After his death in 670, reports of miracles attributed to him and his relics, primarily the ring and his hat, began to spread. He was eventually canonized in 720 by Pope St. Gregory II.

Adapting St. Vincent Lombardi's story to the old Super Bowl Sunday holiday help spread Christianity among the Germanic peoples, but as time passed, his memory, and the memory of the original holiday, faded. As we gather with our friends and loved ones down at the bar or in our living rooms this Super Bowl Sunday season, let's take a moment to reflect on the holiday's origins, and St. Vincent Lombardi's story.

It seems like only yesterday this site only had 15,000,000 comments, but really it was about a month and a half ago. Yesterday, though, this site grew up just a little more and the 16 millionth comment was posted by Lashe. Fittingly, somehow, it was about a candidate.

I missed it yesterday because my parents took me out to lunch for my birthday, so this is a little late. Congratulations, Lashe, and the rest of you who helped push the comment count up with your comments about your cats and bitching about non-preferred candidates. Huzzah!

As I've stated elsewhere on the site recently, I had to adjust the archived age of diaries and comments recently to keep the site running in tip top shape. Unfortunately, those weren't the only things causing some problems - one big one was mojo recalculation. In the spirit of keeping the site running well, I've made some tweaks to how that's done that include less punishing queries and being able to look at your comments in the archived database. I'd like to hear if it's working well.

The bonus goodie for the techies out there: Over the weekend, I dusted off my C skills (which lie sadly neglected out in the garage most of the time) and hacked up lighttpd's mod_magnet module to include memcached support. While mod_cml does include memcached support, it only lets you put the version of a page into memcached, not the content. Now, the new lighttpd front end of Daily Kos has a Lua magnet script runs if the page is visited by an anonymous visitor and checks to see if that page's contents are available in memcached. If it is, then lighttpd returns that content to the anonymous visitor. If not, it continues through to Scoop. In the meantime, Scoop is putting pages for anonymous users into memcached that time out after 2 minutes so they don't get too stale. Thus, anonymous traffic is almost entirely handled by lighttpd, reducing strain on Scoop considerably. As soon as things calm down a little, I'll clean up the patch for it (and the patch for mod_deflate that I updated at the same time) and send them along to the lighttpd developers.

I was making some more improvements to DailyKos this evening while I was putting my daughter to bed, and one of the things I did was fix the caching stuff with a function that returns a UID from the nick provided. Unfortunately, while I was doing this, I managed to make a little mistake and leave a ~ out. To wit:

$snick = s/ /_/g;

vs.

$snick =~ s/ /_/g;

While I of course tested this out on my dev box here at home, I managed to overlook it because I'm the only person who's ever on that box, and sometimes stuff won't turn up there that turns up on the live site. Because of that lack of tilde, everyone was getting the same uid returned when that function ran. It was only like that for about 10-15 minutes, but as far as I can tell everyone noticed.

My apologies for screwing up so royally, and thank you for your kind patience. We're fine now.

Everyone wishes they could read tea leaves or goat entrails and foresee the outcome of the primary. Fortunately for those of us who can't, Nostradamus himself wrote about the primary battles of the 2008 election cycle hundreds of years ago for our benefit. I think after a careful reading of the prophecies, you'll find ample support for your candidate's winning the race.

Comment #15,000,000, she is posted. Congratulations, Seattle Liberal, on your achievement, such as it is.

Figuring out comment #15 million was easier than similar comments in the past, since as part of the massive internal overhaul that I've been working on lately, comments have an auto incrementing integer as their primary key instead of a combination of the 20 byte varchar sid and an integer cid for the comment number. It still uses those for unique keys, and I haven't found a way to make backwards compatible URLs using the new ids for the comments yet (or, most importantly, to use them with the ajax update stuff), so you may or may not ever actually see them.

Again, congrats Seattle Liberal, from your comrade in Olympia Tacoma!

In the interest of preserving civil discourse and honesty in diaries on Daily Kos, the following filters are going to be added to comments and diaries posted on the site.

1. If the filter detects that you are using genocidal terms to describe something that is not, in fact, genocide, the software will automatically tone down your rhetoric to a more appropriate level. For instance, if you write a diary comparing the roving gangs troll rating your comments down to the Ton Ton Macoute, Scoop will automatically rewrite your statement to reflect reality; namely, you're getting pissy because you're posting crap comments and people are calling you on it.
2. Any use of the word "fascist" or "fascism", when not used to describe actual fascists, will be changed to "Whigist" and "Whigism". It's just about as appropriate. Note: for the purposes of this filter, Franco, Salazar, the German-American Bund, and Oswald Mosely are acceptable as fascists, as are groups like the American Nazi Party and other NSDAP inspired groups. Republicans, both neo- and paleo-conservative, the State of Israel, and that cop who confiscated your beer when you were 16 are not.
3. Any use of the term "Orwellian" when not used to discuss the works of George Orwell will be replaced with "I am a fucking idiot who's too lazy to do his own thinking about past, present, and future political situations."

I hope these changes will be a service to you all.

Congratulations, Bill Prendergast, for posting the 14,000,000th comment Daily Kos (as a Scoop site - I don't know how many comments the MT version had) on Tuesday, October 9th. I just realized that it would probably be happening soon, and as soon as I checked I saw that I had missed it by a few days.