The dangers of teaching American exceptionalism

February 21st, 2015 11:36

I believe that the United States is an exceptional country. Not only are we #1 in several key metrics like the world’s largest economy (for now), the largest military, and countries with manned landings on the Moon (#1 and only!), but we are also historically responsible for modern republican democracy itself. We, along with the subsequent French revolution, changed the course of history by pioneering a new form of government that had only ever been hinted at back during the classical era, and proved its superiority with our resounding success. Anyone who tries to downplay the importance of the United States on modern world history simply isn’t paying attention.

But I’m not here to brag on American exceptionalism, and god do I even hate that phrase. Outside of a historical context, even emphasizing it accomplishes little good. This is why I’m dismayed at a recent law proposed by Rep. Peter King that was passed in Oklahoma that bans Advanced Placement History classes due to quibbles over their curriculum for insufficiently whitewashing American history. It’s terrible for all the smart kids in Oklahoma that will not be able to take excellent classes and then get credit for them in college, but there’s something even worse at play.

You cannot teach a perverted version of history. You will be doomed to repeat past mistakes and atrocities if you do. The United States may be exceptional, but we also have a good share of blemishes as well, including slavery, lack of civil rights for non straight white land-owning men, our conquest and subjugation of the native population, and many others. We’re not worse than a lot of other countries in this regard, but we certainly aren’t better, and rejecting a curriculum because it has an even-tempered approach toward history instead of a rah-rah go-America boosting one is negligently short-sighted.

Everyone knows what happens when you spoil kids and tell them they can do no wrong: they turn into monsters with no ability to self-reflect and no compunction against committing evil. Similarly, and this is an issue very close to me personally, everyone knows what happens when you praise kids for being very smart, and emphasize the importance of innate intelligence over diligent study and hard work. So why should it come as a surprise to anyone that when you drill American exceptionalism into kids’ heads over and over again, they come out of it with the belief that America can do no wrong? It’s easy to minimize any historical wrongdoing when you fervently and uncritically know that your country is number one, because really, how bad can slavery be if we did it, and it was part of getting us to where we are now, on top?

We need a more measured sense of introspection than that. Not everything that we did in the history of our country to get to this point was necessary or justifiable. The point is to learn from those mistakes and make damn sure that they never happen again, an attitude which is impossible to adopt if you never learn about those past abuses at all, or are taught exceptional rationalization skills from a young age to paper them over. The bad parts in American history need to be especially emphasized, not ignored, so that particular importance is placed on avoiding repeats. It’s easy to justify any wrongdoing going forward if you don’t recognize those wrongdoings of the past and thus make no effort to be any better in the future.

It’s no accident that the people most fervently pushing a white-washed version of history are the same ones supporting our most egregious ongoing abuses and inequalities, including unjustifiable wars, torture of prisoners, discrimination against homosexuals, removal of the voting rights of black Americans, uncritical support of the police even in cases of extreme unnecessary force, encroachment by religion on secular matters of the state, support of draconian drug policies that lock up millions of Americans for non-violent drug offenses to no purpose, and an unwillingness to help the members of society that are less well-off even though doing so makes everyone better off in the long run. But if you study history, and see what these kinds of policies led to in the past, it’s much harder to support them in the present. Avoiding this is where the movement to teach (dare I say brainwash) American exceptionalism in schools has its ultimate roots. That is why I can never support it.

Python’s negative list indexing is confusing and shouldn’t be taught to novice programmers

March 8th, 2014 13:02

I’ve been programming for a long time, in a wide variety of languages, but never did I see negative list indexing until I learned Python nine years ago to play around with making automated edits on Wikipedia. For those who don’t know, negative list indexing allows you to index into a list from the end instead of the beginning. Whereas arr[0] returns the first item in a list, arr[-1] returns the last item in a list, arr[-2] the penultimate item, etc. It seemed like such a no-brainer; why haven’t all languages implemented this feature? It’s useful, right? That’s what I thought until I recently interviewed a spate of internship program candidates for the upcoming summer, most of whom are learning programming in Python, and I saw firsthand the confusion that negative list indexing causes in novices developers.

I do a fair amount of interviewing at my job, with an average of three or so interviews per week. We’re more often hiring developers than not. Now that I’m leading a development team, I have a huge vested interest in finding the best possible talent because inferior developers directly affect the quality of my team’s output, thus making me look bad. The main thing I focus on during interviews is programming problems. You’d be amazed at how many people do just fine in the talking portion but then fail miserably when asked to actually code something. So I use a progression of increasingly difficult programming problems, the first of which is usually “Write a function that takes a sentence and a number as input, and returns the re-arranged sentence with the specified number of words moved from the back to the front”. So for the example sentence “That is a dark brown fox” and the number 4, the output would be “a dark brown fox That is”. I ask the interviewees to solve the problem in whatever language they’re most comfortable with, because I’d rather get good developers in general than someone with certain specific language skills.

A typical solution in C#, which I get a plurality of responses in, looks like this:

public string Rearrange(string sentence, int numWords)
    string[] words = sentence.Split(' ');
    string answer = "";
    for (int i = words.Length - numWords; i < words.Length; i++)
        answer += words[i] + " ";
    for (int j = 0; j < words.Length - numWords; j++)
        answer += words[j] + " ";
    return answer.Trim();

It's not an amazing solution; there's a faster and more efficient way to do it that doesn't require splitting up the input into a words array at all (I'll leave that as an exercise to the reader). But it gets the job done, and most people end up thinking about the problem in this way. Yet a curious thing happened when I started interviewing current undergraduate students for the intern positions: most of them ended up using Python, and all of Python solutions used negative list indexing and ended up going off the rails because of it. I understand why they immediately thought to use negative indexing; we are asking them to grab things off the end of a sentence, after all. But it consistently sowed confusion, and never clarified.

There were two main problems that I saw when people used negative list indexing. One problem was that the interviewees would start with -1 and then decrement to -2, then -3, etc. This seems logical, but you end up reversing the order of the words in the rear segment, such that your output sentence from the above example becomes "fox brown dark a That is". What you really need to do is count "down" through the words in reverse, in the order -4, -3, -2, -1. But that's a bit harder to wrap your mind around than avoiding negative indexes entirely and simply counting up through the words in the order 2, 3, 4, 5. In particular I saw a lot of reversed parameters to the range function, which ends up executing the inside of the for-loop zero times. The problem looks like it's dealing with the end of the list, but it's really not; it's dealing with the middle of the list and counting up through to the end of it, for which using positive indexes is a lot more handy. But as soon as you say "end" to a novice programmer who's learned the "trick" of negative list indexes, that's where their mind immediately goes, and they tend to get stuck.

The second main problem with negative list indexing is that it doesn't mesh well with positive list indexing, both of which you end up needing to use to solve this problem. For instance, [0] is the first element in a list, and [1] the second, but [-1] isn't the second-to-last element in the list; it's the last one. This caused a lot of confusion. Positive list indexes are zero-indexed but negative list indexes are one-indexed. Fencepost errors abounded, or what you might even call reverse-fencepost errors, where interviewees expressed the index of the start of the second segment as either [-numWords +1] or [-numWords +-1] (both wrong but in different ways) when it should simply be [-numWords]. The interviewees simply didn't have enough experience using the two together to understand how they worked in unison, whereas when I prompted them to write the boundary conditions of both for-loops using positive indexing, the mental confusion usually went away and they were able to come up with a workable solution.

For the record, here is a correct solution in Python that uses negative list indexing. It doesn't look that tricky when you see it presented as an answer, but trust me when I say that no one was coming up with it on their own without access to a Python interpreter. The boundary condition of 0 on the first range in particular was highly un-intuitive to all of the interviewees.

def rearrange(sentence, numWords):
    words = sentence.split()
    answer = ""
    for i in xrange(-numWords, 0):
        answer += words[i] + " "
    for i in xrange(0, len(words) - numWords):
        answer += words[i] + " "
    return answer.strip()

And there's one final point that I can't help but bring up: reverse traversal of lists is cache inefficient. Granted it doesn't matter for this problem, and it probably won't matter for most things you do in Python, but if you're trying to write highly performant code in C/C++, like for scientific computation or a game engine, you're going to cause a lot more cache misses by trawling through memory in reverse order.

And the spammers have won (comments are now disabled)

December 13th, 2013 00:01

The problem with spammers on the web that I talked about before has continued getting worse and worse, to the point where hundreds of spam comments (and no identifiably non-spam comments) have made it through onto this blog. Akismet wasn’t doing the job, and I don’t maintain this blog often enough to keep all of the spam out. My only fix is to disable all comments. It sucks, because the two defining features that characterize a blog are (a) being personal and opinionated and (b) allowing multi-way communication between the author and the readers, and the readers with each other. I’m no longer doing half of that, so in some very real senses this is no longer a blog (though feel free to still email me if you want to chat). I’m really sad about it. But the spammers have won. It’s no longer feasible to try to maintain your own blog out there. Just host it with WordPress or similar and let someone else handle the spam problem, because I can’t stay on top of it any longer.

Using LINQ in C# to easily get a list of most often used words

May 20th, 2013 15:26

A pretty common programming interview question is to parse an input sentence and return the list of unique words used in that sentence. A further elaboration on that problem, the one that this post will be addressing, is to additionally calculate the number of occurrences of each word, and then return the top K words for some input value K. I’ll be demonstrating a simple solution to that problem in C#, both because I’ve been using it a lot recently and also because the choice of C# gives us access to LINQ, which is a powerful C# language feature that allows queries on collections using a SQL-like syntax. The top K problem is incredibly easy to solve in SQL, and boils down to SELECT TOP @K ..... FROM Words ORDER BY Word.Name DESC. The C# solution is similarly easy.

First, I’ll clarify the assumptions that I’m using (and that would be wise for an interviewee to address if none of these are made explicit):

  • I’m writing my solution to be case-insensitive. “case”, “Case”, and “CASE” will all thus count as the same word.
  • I’m not dealing with ties. If you ask for the top 3 words by occurrence and there are 5 words that are all used the most in equal numbers, then you’re still only going to get three words of those five, selected in no particular manner.
  • I’m going to use the C#’s language spec definition of a non-word character in regular expressions to separate the input sentence into words. The naive solution would be to split the string on only spaces, but then you’re not handling punctuation correctly.

And for the solution:

Dictionary<string, int> GetTopKWords(string input, int k)
	string[] words = Regex.Split(input, @"\W");
	var occurrences = new Dictionary<string, int>();
	foreach (var word in words)
		string lowerWord = word.ToLowerInvariant();
		if (!occurrences.ContainsKey(lowerWord))
			occurrences.Add(lowerWord, 1);
	return (from wp in occurrences.OrderByDescending(kvp => kvp.Value) select wp).Take(k).ToDictionary (kw => kw.Key, kw => kw.Value);

The vast majority of this code is responsible simply for finding the list of unique words the number of occurrences of each one. Once that is known, finding the top K words is a single line of code thanks to the power of LINQ. All we’re doing is ordering the words by frequency in descending order and taking the top K words. Without LINQ, there would be significantly more book-keeping code required to do this, which would make a good exercise for the reader (e.g. solve this problem in Java). The first roadblock you’ll probably run into is that you can’t simply flip the keys and values of the array, because the frequency counts aren’t unique. The best I’ve come up with is to construct a list of ordered tuples out of the dictionary of words, order it on the occurrence count part of each tuple, and then extract the first K elements from the resulting ordered list and return it. Let me know in the comments if you have a better solution.

Oh, and here’s an example input/output for the program, handling the display of output using LINQPad:

var input = "the quick brown fox is brown and jumps over the brown log over the long fire and quickly jumps to a brown fire fox";
GetTopKWords(input, 10);


Top K Words sample run

Fixing an error with being unable to add user fields in Drupal 7

April 4th, 2012 10:59

Drupal 7 has a nice built-in Fields functionality which can be used to add fields to any entity. As applied to users, this replaces the previous Profile module which was used in Drupal 6 to add fields to users. However, right after creating a new Drupal 7 site, I couldn’t figure out how to add fields to users.

I followed these instructions

Just go to:

Configuration > People > Account Settings

Then click on the Manage Fields tab, and then just manage the fields just like you would for a Content Type.

But there wasn’t a “Manage Fields” tab on that page. Going to the URL directly, admin/config/people/accounts/fields, was redirecting me back to the Account Settings page. After banging my head against the wall for ten minutes I finally realized that the reason I couldn’t see this tab was because the “Field UI” module wasn’t enabled. Go to the List Modules page, enable that module (and its dependencies), and now you should be able to add fields to users.

The many ways in which Gravity’s Rainbow directly inspired Neon Genesis Evangelion

January 19th, 2012 22:46

I recently read the well-known post-modernist novel Gravity’s Rainbow (1973) by Thomas Pynchon, and I noticed a number of surprising similarities between it and the well-known Japanese anime series Neon Genesis Evangelion (1995). In fact, there are so many similarities between the two, both thematically, stylistically, and plot-wise, that I am forced to conclude that Hideaki Anno, the writer and director of Evangelion, must have read Gravity’s Rainbow and drew upon it specifically for inspiration in creating his series. Unfortunately, I haven’t found any discussion of the similarities between these two works, hence the need for this post.

I’ll assume a familiarity with Evangelion for the remainder of this post (which allows me to focus on summarizing and explaining Gravity’s Rainbow). There are some spoilers for Gravity’s Rainbow below if you haven’t read it yet, but the novel is so meandering and expansive that it’s not possible to ruin it.

I’ll examine the thematic similarities first. Gravity’s Rainbow runs from the beginning of World War II, through V-E Day, and through the occupation of Germany by the Allied forces. The first part of the novel, which takes place in England during the German bombing and rocket campaign, takes place under a heavy siege mentality. V-2 rockets are falling often, at random, and killing lots of civilians. There are many scenes that take place deep within bunkers, or have military personnel travelling to the scene of the latest rocket strike to investigate the effects. This whole section of the novel feels very similar to the overall mood in Tokyo-3 as Japan is besieged by one attacking Angel after another, right down to the missions being ordered from within the safety of a bunker.

Gravity’s Rainbow is suffused throughout with the paranormal, the occult, the bizarre, and many different references to psychology (especially that of Sigmund Freud). Pynchon is every bit as obsessed with psychology as Hideaki Anno, to the point where if you couldn’t handle the original last two episodes of Evangelion, you probably won’t enjoy the similar parts of Gravity’s Rainbow either, as there is an equal amount of psycho-analytical musing in it. Both works examine military hierarchies and point out some of the inherent absurdities in them. Gravity’s Rainbow especially focuses on jargon-heavy, acronym-laced, secret military, espionage, and industrial research organizations, and the interplay and conflict between them — exactly like Evangelion.

Gravity’s Rainbow is also laced throughout with sexuality and sexual deviance, which is another theme that Evangelion explores quite thoroughly. A pervading sense of paranoia is present throughout the text. It has a large amount of technological detail in it, verging towards technobabble on many occasions, same as Evangelion. It even has one particularly memorable scene in which a boy is plugged into a harness made out of Imipolex G, an erectile plastic polymer (you can’t make this stuff up) that interfaces directly with the boy’s neural network. This harness is then put inside a V-2 rocket and launched with the boy as the unwitting cargo (not as pilot). Shades of Shinji being plugged into an Evangelion and then losing control of it, anyone?

But the most convincing case of Evangelion’s inspiration from Gravity’s Rainbow can be made by looking directly at some examples from the text of the novel. I’ll present three passages that were so startlingly similar to Evangelion that I set the book aside in amazement long enough to take notes, wondering how nobody had ever caught this before (or, at least, if they did, why they didn’t post their findings online). All page numbers I’ll be using are from the original 760-page edition of Gravity’s Rainbow.

First, we’ll start with a passage from page 151, in which a Royal Air Force bomber squadron is attacking the German town of Lübeck.

It’s a dangerous game Cherrycoke’s playing here. Often he thinks the sheer
volume of information pouring in through his fingers will saturate, burn him out
. . . she seems determined to overwhelm him with her history and its pain, and
the edge of it, always fresh from the stone, cutting at his hopes, at all their hopes.
He does respect her: he knows that very little of this is female theatricals, really.
She has turned her face, more than once, to the Outer Radiance and simply seen
nothing there. And so each time has taken a little more of the Zero into herself.
It comes down to courage, at worst an amount of self-deluding that’s vanishingly
small: he has to admire it, even if he can’t accept her glassy wastes, her appeals
to a day not of wrath but of final indifference. . . . Any more than she can accept
the truth he knows about himself. He does receive emanations, impressions . . .
the cry inside the stone . . . excremental kisses stitched unseen across the yoke of
an old shirt. . . a betrayal, an informer whose guilt will sicken one day to throat
cancer, chiming like daylight through the fourchettes and quirks of a tattered
Italian glove . . . Basher St. Blaise’s angel, miles beyond designating, rising over
Lübeck that Palm Sunday with the poison-green domes underneath its feet, an
obsessive crossflow of red tiles rushing up and down a thousand peaked roofs
as the bombers banked and dived, the Baltic already lost in a pall of incendiary
smoke behind, here was the Angel: ice crystals swept hissing away from the back
edges of wings perilously deep, opening as they were moved into new white
abyss. . . . For half a minute radio silence broke apart. The traffic being:

St. Biaise: Freakshow Two, did you see that, over.

Wingman: This is Freakshow Two—affirmative.

St. Biaise: Good.

No one else on the mission seemed to’ve had radio communication. After
the raid, St. Biaise checked over the equipment of those who got back to base
and found nothing wrong: all the crystals on frequency, the power supplies
rippleless as could be expected—but others remembered how, for the few
moments the visitation lasted, even static vanished from the earphones. Some
may have heard a high singing, like wind among masts, shrouds, bedspring or
dish antennas of winter fleets down in the dockyards . . . but only Basher and
his wingman saw it, droning across in front of the fiery leagues of face, the eyes,
which went towering for miles, shifting to follow their flight, the irises red as
embers fairing through yellow to white, as they jettisoned all their bombs in no
particular pattern, the fussy Norden device, sweat drops in the air all around its
rolling eyepiece, bewildered at their unannounced need to climb, to give up a
strike at earth for a strike at heaven . . . .

Group Captain St. Biaise did not include an account of this angel in his official
debriefing, the W.A.A.E officer who interrogated him being known around the
base as the worst sort of literal-minded dragon (she had reported Blowitt to
psychiatric for his rainbowed Valkyrie over Peenemünde, and Creepham for
the bright blue gremlins scattering like spiders off of his Typhoon’s wings
and falling gently to the woods of The Hague in little parachutes of the same
color). But damn it, this was not a cloud. Unofficially, in the fortnight between
the fire-raising at Lübeck and Hitler‘s order for “terror attacks of a retaliatory
nature”—meaning the V-weapons—word of the Angel got around. Although
the Group Captain seemed reluctant, Ronald Cher-rycoke was allowed to probe
certain objects along on the flight. Thus the Angel was revealed.

The similarities to Evangelion here are obvious. The bomber squadron runs into an apparition in the sky, while all radio contact goes dead. This apparition is even known as the Lübeck Angel.

Next, we have a small fanciful vignette from page 674:

Onward to rescue the Radiant Hour, which has been abstracted from the day’s
24 by colleagues of the Father, for sinister reasons of their own. Travel here gets
complicated—a system of buildings that move, by right angles, along the grooves
of the Raketen-Stadt’s street-grid. You can also raise or lower the building itself,
a dozen floors per second, to desired heights or levels underground, like a
submarine skipper with his periscope—although certain paths aren’t available to
you. They are available to others, but not to you. Chess. Your objective is not the
King—there is no King—but momentary targets such as the Radiant Hour.

Notice that the city of Raketen-Stadt as being described here is pretty much identical to Tokyo-3, in which the skyscrapers are above-ground during the day, but are lowered underground into the Geo-Dome at night or when the city is under attack from Angels.

And finally, from page 753 near the very end of the novel:

The countdown as we know it, 10-9-8-u.s.w., was invented by Fritz Lang
in 1929 for the Ufa film Die Frau im Mond. He put it into the launch scene to
heighten the suspense. “It is another of my damned ‘touches,’ “ Fritz Lang said.

“At the Creation,” explains Kabbalist spokesman Steve Edelman, “God
sent out a pulse of energy into the void. It presently branched and sorted into
ten distinct spheres or aspects, corresponding to the numbers 1-10. These are
known as the Sephiroth. To return to God, the soul must negotiate each of the
Sephiroth, from ten back to one. Armed with magic and faith, Kabbalists have
set out to conquer the Sephiroth. Many Kabbalist secrets have to do with making
the trip successfully.

“Now the Sephiroth fall into a pattern, which is called the Tree of Life. It
is also the body of God. Drawn among the ten spheres are 22 paths. Each path
corresponds to a letter of the Hebrew alphabet, and also to one of the cards called
‘Major Arcana’ in the Tarot. So although the Rocket countdown appears to be
serial, it actually conceals the Tree of Life, which must be apprehended all at
once, together, in parallel.

“Some Sephiroth are active or masculine, others passive or feminine. But the
Tree itself is a unity, rooted exactly at the Bodenplatte. It is the axis of a particular
Earth, a new dispensation, brought into being by the Great Firing.”

“But but with a new axis, a newly spinning Earth,” it occurs to the visitor,
“what happens to astrology?”

“The signs change, idiot,” snaps Edelman, reaching for his family-size jar of
Thorazine. He has become such a habitual user of this tran-quilizing drug that
his complexion has deepened to an alarming slate-purple. It makes him an oddity
on the street here, where everybody else walks around suntanned, and red-eyed
from one irritant or another. Edelman’s children, mischievous little devils, have
lately taken to slipping wafer capacitors from junked transistor radios into Pop’s
Thorazine jar. To his inattentive eye there was hardly any difference: so, for a
while, Edelman thought he must be developing a tolerance, and that the Abyss
had crept intolerably close, only an accident away—a siren in the street, a jet
plane rumbling in a holding pattern— but luckily his wife discovered the prank
in time, and now, before he swallows, he is careful to scrutinize each Thorazine
for leads, mu’s, numbering.

“Here—” hefting a fat Xeroxed sheaf, “the Ephemeris. Based on the new

“You mean someone’s actually found the Bodenplatte? The Pole?”

“The delta-t itself. It wasn’t made public, naturally. The ‘Kaisers-bart
Expedition’ found it.”

A pseudonym, evidently. Everyone knows the Kaiser has no beard.

This illustrates many thematic similarities with Evangelion, including references to the Kabbala, the Tree of Life, mythological angels, the occult, cataclysmic events, and even a search for The Pole (ahem, Second Impact). And note that it takes place in the context of a long-winded, jargon-heavy discussion between military figures. If you animated this passage it would fit right into an episode of Evangelion.

I believe that the similarities between Gravity’s Rainbow and Evangelion (which came out two decades later) have been established beyond a reasonable doubt. I wouldn’t go so far as to use the word “steal”, but in my mind, Evangelion directly owes a lot of its feel and setting to Pynchon’s work. As a consequence of this, if you’re an Evangelion fan, you owe it to yourself to read Gravity’s Rainbow. Not only is it a good novel in its own right, but by reading and understanding some of the inspiration behind Evangelion, you’ll get a better understanding of Evangelion itself.

Long time no blog

February 20th, 2011 18:39

It’s been a long time since I’ve posted anything here — over a year, anyway, which might as well be a decade in Interwebs time. The funny thing is that this blog gets more views now than it ever did before. That’s right, the Google searches leading to the backlog of old content have continued to drive traffic in ever increasing numbers even though nothing new has been posted in awhile. As a result of that, the AdSense ads on this site continue to support the cost of server hosting; hence why the site is still around even though I haven’t written anything new in awhile. Now I’m not making any money off it, but it’s always nice to have access to a co-hosted GNU/Linux server running somewhere (especially when said access is free).

One wonders what traffic on this site might be like now if I had kept updating it regularly, but whatever. I used that time for other things. Like reading!

I’ve been working my way through Modern Library’s Top 100 English Novels of the 20th Century. I’ve also been taking many a detour through other classic American novels that didn’t make the board’s list (To Kill A Mockingbird anyone?) and that I never got a chance to read in high school or college.

Most recently I read John Steinbeck’s Cannery Row. I’m going to have to go back and read a bunch more of his books (the only other one I think I’ve ever read was The Grapes of Wrath, back in high school). I really enjoy him as an author, especially the little textural vignettes that have nothing to do with the main story. Unbelievably, one of my friends recently cited these interesting interruptions as a reason why they didn’t like Steinbeck. Crazy talk.

Also, Ernest Hemingway has been very enjoyable. I really appreciate what he’s able to convey using such few words. I know I can write that simply, but my icebergs wouldn’t have anything below the surface of the waves.

Reminiscing about the naïve, spam-free days of the web

July 21st, 2009 23:48

Remember a long time ago when the web was free of spam? I’m not talking about email, which has had spam problems for awhile, I’m talking about the web. Nowadays, the web is festering with link-crawling spambots. Anyone with a blog, Twitter account, or heck, even a webpage with a simple submit form with some text fields on it, knows this. There’s not much that can be done about it besides spam-detection heuristic algorithms and CAPTCHAs.

Well, I just recently found some code that I wrote way back in 2002 that displays a blissfully unaware naïvité of what was to come. That code was part of my website Fyre’s Domain, which I have since put an archived copy of online. I had just been learning Perl CGI and I wanted to write a simple guestbook/comments form that readers could use to give me feedback without having to use a mailto: link. This was in the era before blogging software was commonplace — what I was running was a home-brew blog, but before the word “blog” was even invented. I basically copied the format from one of the first chat rooms I ever used, Paddynet, way back in 1995 or so. The “chat room” consisted of an HTML form that would dump unvalidated input (including HTML tags) into a chat buffer displayed on the page that would display the last 30 or so messages.

Paddynet was around long before spambots, but my site was started right when they began appearing in the wild, and the code proceeded to run for another 7 years until I just shut it off.

You can probably guess what happened.

The only reason I even re-discovered this code is because I happened to notice it was getting an unusual number of hits in my web analytics software. And those hits were anything but benign. My poor naïve Perl CGI comments submission form has accumulated 26 MB worth of text over the years, all of it spam. And since I figure it may be interesting to someone to see exactly what seven years of web spam looks like, you can download it for yourself (a text file thankfully compressed down to just 1.8 MB). If anyone finds any interesting trends in popular spam topics over the years in there, do let me know.

So those are the dangers of trusting user input on the web these days. Revel in the blissful simplicity of the following code, which was all it took to implement a comment submission system back in the day. Nowadays you couldn’t get away with anything even close to it. As my data proves, you’ll be eaten alive by spambots.


use CGI qw(:standard);

print header;
print start_html('Leave Comments on Fyre'),
	h1('Leave Comments on Fyre'),
	"<i>Note, all fields are optional, but empty comments will be ignored.</i><br>", 
	"Name: ", textfield(-name=>'name',-default=>''),
	"E-mail: ", textfield(-name=>'e-mail',-default=>''),
	"Your Comments: <br>", textarea(-name=>'Comments',-rows=>10,-columns=>50,-default=>''),
	submit('Submit'), reset,

if (param() && param('name') ne '' && param('Comments') ne '') {
	$date = `date '+%H:%M %m/%d/%Y'`;

	print '<i>Your comment has been posted.</i><hr><br>';
	@foo = "\n\n" . '<br><b>' . param('name') . '</b> ' . "\n" .
	'<u>' . param('e-mail') . '</u> ' . "\n" . '<i>' . $date . '</i>' . 
	"\n" . '<table><tr><td width = "100%">' . param('Comments') . 
	push @foo, `cat mk.txt`;
	open CFILE, ">mk.txt" or die "Failed to open comments file!";
	print CFILE @foo;
	close CFILE;

@foo = `cat mk.txt`; print @foo;

print 'This program is open source, and released under the GPL by Ben McIlwain, 2002.  See 
the source <a href = "mk_script.txt">here</a>.';
print end_html;

Right-wing terrorism

June 10th, 2009 16:26

Today, an anti-semitic terrorist attacked the Holocaust Memorial in Washington D.C. (I’ve been there, and yes, one visit is enough for a lifetime). Last week, an anti-abortion terrorist assassinated a doctor.

Why is the media so afraid to use the word “terrorist” to accurately describe right-wingers engaged in the act of terrorism? Is it that whites can’t be terrorists? Only Arabs?

Until we call it what it actually is, we can’t address it properly.

And since right-wingers were so keen on using water-boarding against terrorists, do you think they’d mind if we tortured these home-grown right-wing terrorists?

My once-tiny GNU/Linux desktop morphs beyond all recognition

April 14th, 2009 19:30

Enermax Chakra
Almost a year ago, I bought a cute little desktop from Dell with the intent of using it as a GNU/Linux desktop alongside my existing Windows desktop. Its name is Vertumnus. But things don’t always turn out as planned. I quickly started using Vertumnus as my exclusive desktop PC, booting the Windows machine only to play games. Eventually I reformatted the Windows computer and the only applications I’ve reinstalled have been games, so it’s pretty much reduced to a gaming appliance at this point, like an XBOX360 but better.

The only problem is that when I originally bought Vertumnus, I didn’t have all of this in mind, and so I bought it rather under spec. I would’ve been better off just buying a better computer from the get-go. As a result, I’ve had to do quite a few upgrades over the past year to get it to meet my needs. From the very beginning I added more RAM and another hard drive. Then it joined a Stand Alone Complex. Then I added another hard drive. From the outside it still looked the same, but a lot of the interior was upgraded. Now even that is no longer true.

Yesterday, I spent two hours (and another $160) redoing the computer even further. The case was too cramped and was preventing further upgrades. So I moved the computer into a new case, the Enermax Chakra. It’s appreciably bigger than the previous Dell case. It’s also a lot more flexible on the inside in terms of which parts will fit into it. Why the Chakra? I only had two criteria, but the Chakra was pretty much the only case that met both of them: 1) It had to have a 250mm fan, but 2) No LEDs. Both criteria come from my computer living in my bedroom: it has to be silent (hence a big, slow-spinning fan) and it has to be dark, so that I can sleep!

Since the case didn’t come with any fans besides the huge 250mm one, I purchased two of the quietest 120mm fans in existence, the Scythe Gentle Typhoon. Again, my criteria were the same: Quiet and no LEDs. The Gentle Typhoons best met those. I also had to get a new power supply, because the 250 Watt one from Dell isn’t able to accommodate the video card I was about to put in. So I went with the Corsair 550W PSU. It was the power supply that best met my criteria: High efficiency (85%!), quiet (a big 120mm fan), and no LEDs. And it’s more than enough to power the video card that I put in, a hand-me-down GeForce 8800 GTS. Yes, that’s right, I finally got tired of the inferior performance of the Intel integrated graphics card. Now I can actually play modern 3D games in GNU/Linux.

And as if all that wasn’t enough, while transitioning all of the parts from one case to another, the CPU fan developed a faulty bearing which makes it obnoxiously loud. So the first thing I hear upon starting up my supposed-to-be-silent computer is a loud whirring fan noise. Rather than giving up my dreams of a silent computer, I ordered a replacement CPU fan/heatsink, the Arctic Cooling Freezer 7 Pro. Why that one? I already have one in my Windows computer and it cools really well. Plus it’s quiet. It hasn’t arrived yet, but it’s going into Vertumnus as soon as it does.

The new GeForce 8800 GTS is so large that it covers up one of the SATA ports on the Dell motherboard (and another one is rendered inaccessible to all but right-angle SATA connectors). Since I have three SATA hard drives and one SATA DVD-R drive, that’s a problem. The DVD drive is currently unplugged, but I’ll swap it out for an IDE DVD-R drive from my Windows desktop soon — thankfully, the video card doesn’t block the IDE port.

Once all of this is done, the only original parts that will remain in Vertumnus from the original purchase will be the Intel Core 2 Duo E7200 processor, 2 1 GB sticks of DDR2 RAM, the motherboard, and one 500 GB hard drive. And that’s after less than one year. Clearly, I tried saving too much money by buying a system far below my ultimate desired specifications, then wasted a bit more than those savings on upgrades. And I can’t even say the upgrades are done. At some point I’m going to need another hard drive, but since I’m all out of SATA ports, I’ll either have to get an add-in card or replace the motherboard. The original RAM that Dell shipped was pretty slow, and can easily (and cheaply) be replaced with something better. And the processor is looking slightly anemic. A nice quad-core processor would be fun to play around with …

Long story short, in another year, it’s quite possible that the only component remaining from my original purchase will be the 500 GB hard drive and a SATA cable or two. I guess I learned my lesson. Don’t try to save too much money on a computer if, at heart, you’re really just a techie who demands performance.