Cyde Weys Musings

I’m doing some research on Unicode and compression algorithms right now for a side-project I’m working on, and I came across a highly ranked Google search result for a UTF-8 munging code snippet that is so idiotic I couldn’t let it pass without comment. If this post helps even one person who would’ve otherwise followed the linked advice, it is worth it.

First, some background. UTF-8 is a character encoding format that can pretty much handle any character under the Sun, from the English alphabet to Japanese kanji to obscure extinct languages. It even includes thousands of esoteric symbols used in smaller fields of study that you’ve probably never even heard of before. But the nice thing about UTF-8 is that it is variable-length. Standard ASCII characters (including everything on a standard English keyboard) only take one byte to represent. All of the common characters from other widely used languages typically take just two bytes to encode. It’s only the really obscure characters that require more than two bytes.

So now you see why the linked “solution” is so stupid. This guy says he is “designing a little client/server binary message format” and wants “a simple and quick way to encode strings”. Well, duh — use UTF-8, no ifs, ands, or buts about it. It’s simple, quick, and already implemented in any programming language you can think of, so it requires no additional coding. There are all sorts of dumb ways to unnecessarily reinvent the wheel in sotware engineering, but trying to come up with your own character encoding is particularly idiotic. It’s really tricky to get right because there are so many corner cases you’ll never even know existed until they cause your application to break. The Unicode Consortium exists for a reason — what they do is hard.

This guy even confesses that his expected input will probably not contain Unicode characters that are longer than 2 bytes. So there is no justification at all for what he does next — he creates a mangled version of UTF-8 that turns all Unicode characters 3 bytes and longer into question marks, instead of just leaving them as is. So instead of allowing a rare character to take an additional byte or two, it gets mangled. And to accomplish this, he has to create his own custom encoding solution that is an intentionally broken version of UTF-8. That’s the worst part — he’s wasting time creating completely unnecessary code, that will need to be maintained, that will need to be debugged — and for what?

Of course, none of the people responding to his thread point out that what he is trying to do is stupid. They just smile and hand him some rope.

I’m up in Parsippany, New Jersey at the moment on business travel. That in itself wouldn’t be anything special, except that the eastern seaboard was just rocked by a huge snowstorm. I had to leave a day early to ensure that I made it here for an important meeting Monday morning, only for that meeting to be canceled while I was en route and incommunicado. To add insult to injury, none of the client employees I work with even showed up for work today, and my car died at the hotel this morning so I walked to the client site in the falling snow. And just for some added excitement, I had to run to escape the torrent from an oncoming snowplow at one point.

The drive up here was no picnic either. About an hour in it started raining, then quickly turned to snow. Thankfully none of it started sticking to the road until I arrived at the hotel four hours and many wrong turns later (not the best time to try a new route). I saw a surprising number of other vehicles driving in the snowstorm without lights on, including one semi-trailer which kept on disappearing and re-emerging from the mist of snow in a terrifying fashion. Even my high beams didn’t provide nearly enough illumination to see the road ahead of me. This was made worse by the constant glow of headlights shining over the jersey barrier from vehicles in the opposite direction, like some dividing line across the horizon, which lit up the entire sky from about six feet above the road on up. The road was thus made darker and less see-able by contrast.

The only thing that hasn’t sucked about this trip so far is ham radio. Sunday night is an excellent time to work the ham bands, which is what I spent my whole commute doing. Repeater contacts have become passé for me these days because they are so easy; at any random point along the I-95 corridor, you can hear at least a couple simultaneous conversations on various nearby repeaters. As such, I focus mostly on making simplex (direct) contacts, which at least provides somewhat of a challenge, especially while operating mobile. I was mostly using the national calling frequency on 2 meters, which is 146.520 MHz, though I did talk to one man on another simplex frequency while idly scanning the band.

I made more simplex contacts during this trip than I ever have before. At one point I was talking with two to three people simultaneously, a feat I’ve never experienced outside of pre-arranged simplex nets while operating stationary. I had some pretty long conversations with stationary operators, as well as some shorter conversations with other mobile operators (as mobiles tend to be a lot more limited in terms of antenna size, elevation, and to a lesser extent, transmitting power).

But the neatest point in the trip was when I briefly became the best ham radio station in the whole area.

I had been talking with a stationary operator for around fifteen minutes. The signal went from bad to good to bad as I-95 took me closer and then farther from his position. His signal was never stronger than S-5 (S-meters give a measure of signal strength from 1 to 9, on a logarithmic scale). About ten minutes after we said our good-byes and he faded into the radio-frequency mist, I arrived at the foot of the Delaware Memorial Bridge.

All of a sudden, the stationary operator I had been talking to earlier came in again. And his signal strength just kept getting better and better. We excitedly traded signal reports in a rapid-fire series of transmissions, remarking on how much the signal quality of the other was improving by the second. My S-meter kept on climbing until it pegged at S-9, still 50 feet shy from the apex of the bridge. The other operator’s signal was full-quieting, meaning that his signal was so strong that not only could I hear him perfectly, even the lulls between the words of his transmission were perfectly silent (because his carrier was so strong that it overwhelmed the ambient radio-frequency noise).

Then as I reached the apex of the bridge, some 200 feet in elevation above the ground and quite the enviable radio location, something really cool happened.

I was able to make contact with my previous contact, much further distant than even the current contact that had just gotten back in range. And in between the gaps in our conversation, I heard a multitude of other voices rising above the static, a chorus of conversations on the calling frequency many miles distant in all directions on the compass rose. So many things were being said at once that I couldn’t make sense of any individual transmission. I could only hear it all as a collective murmur. All of these people out there, each holding separate conversations — and unlike any of them, I could hear it all at once.

As I crested the apex of the bridge, the signal strength from my primary contact rapidly faded back down the S-meter, and with one last hurried transmission, we said good-bye. Then he, along with everyone else, was lost to the static, and I was alone again.

Frank quickened his stride and took a leaping jump onto the waist-high brick wall separating the sidewalk from the field. Kathy rolled her eyes at him. None of the other students busily hurrying to or from their classes registered a response at all.

“You know, this isn’t the playground at kindergarten,” Kathy teased Frank.

“Yeah, I know,” Frank responded. “If it was, I wouldn’t have made that jump.” He cracked what he hoped was his most winning smile. Arms spread wide, he balanced carefully as he continued walking along the brick wall. Kathy walked along beside him.

Kathy turned away and grunted, trying unsuccessfully to hide a smile. On the other side of the wall, a large group of students – too many on the field at once for a real match – was kicking a soccer ball around on the patchy field. They didn’t look especially coordinated, nor was there even a clear delineation between teams, but at least they looked to be having fun.

The two walked awkwardly in silence for awhile, avoiding each other’s gaze, until Frank grew bored of the novelty of walking along the wall and jumped back onto the sidewalk, nearly tripping in the process. He hoped Kathy hadn’t seen that, but of course, she had.

After several more moments of silence, Frank turned to look at Kathy. “Ready for the exam next Monday?” he asked.

“Not at all,” Kathy said mock-indignantly. “And that was a complete waste,” she continued, gesturing behind her. “How come we’re already learning new material that won’t even be on the exam?”

“He’s trying to teach sadism.” Frank paused for a few instants for effect. “By example.”

Kathy smirked. Then, she quickly side-stepped to the left to avoid an oncoming bicyclist on the sidewalk, bumping into Frank in the process.

“Idiot,” she said angrily, looking over her shoulder at the bicyclist. “The sidewalk is for people.”

A car whizzed by on the road to their right, braking and honking angrily at an absent-minded student who had just started jay-walking across the road.

“So, do you want to study for the exam together?” Frank asked. A slight tremble in his left hand would have betrayed his growing nervousness, had anyone noticed it.

“Oh, come on, you ace all the tests,” Kathy said with a tone of feigned annoyance.

“Well, I could help you.”

“Come on, you have better things to do with your time,” she said, smiling.

“Nope, I really don’t,” he said half-jokingly.

“All right, well I hope you don’t expect you’re getting anything out of this,” she said, her smile widening. Frank felt his heartbeat quickening.

They reached a group of students anxiously waiting on a patchy piece of grass between the sidewalk and street for a break in the steady stream of cars. The real crosswalk was a hundred feet up the road, but it didn’t lead directly to the walkway leading between the Mathematics and Physics buildings back to the dorms.

“So, when do you want to meet up?” Frank asked hopefully. A gap in the cars arrived and the group pushed out into the streeth. Frank stepped off the curb and followed them, Kathy closely behind him.

“How about toni-”

The world turned red in an instant.

Jagged loping arcs of pure white electricity lept between the power lines leading to the Physics building.

The red afterglow faded just as soon as it had come.

Everyone stopped in their tracks. One man fell over himself.

An oncoming car plowed into the students in the road behind Frank at low speed, the panicked driver stomping on the brakes to little effect.

Frank pulled Kathy up onto the opposite sidewalk.

People witnessing the carnage behind them began screaming for help. Others just began screaming.

“What the-” Kathy began saying. A loud boom from elsewhere on campus cut her off. The assembled crowd whipped around and looked in the direction of the presumed explosion, but saw nothing. The screaming ceased as people strained to hear anything further.

More out-of-control cars collided into each other on the roads, followed by bewildered motorists stepping out and staring at their vehicles in exasperation.

“Something happened,” Frank said softly. He was still holding Kathy’s hand, but he was no longer thinking at all about how he could turn a study session into something more. The traffic light up the street was dead.

Frank glanced at his watch. It was blank. “Something big happened.”

Kathy fumbled with her cell phone, but it wouldn’t turn on.

One of the students that had been struck by the car unsteadily got to his feet, clutching his leg. He started yelling at the driver of the car, unaware that larger problems were afoot.

“Nothing works!” Kathy said with increasing exasperation. She threw the cell phone down in disgust. The other students were collecting themselves.

“What happened?” Kathy asked Frank. The silence was deafening, punctuated only by curious and frightened whispers. He didn’t answer her right away. He was staring up in the sky, transfixed. Slowly, Kathy followed his gaze, and was frozen in horror too.

“Remember this moment. The moment that everything changed.”

They stared at the jetliner in the sky several miles out. It continued on its steep downwards trajectory, trailing a contrail of smoke that was too gray to be just water vapor. It developed a list to the left, and then with a chorus of shocked gasps from the attentive crowd, it rolled over entirely, and quickened its meeting with the ground. In the distance, in the direction of the airport, two more jetliners sought a similar fate.

The exam that would now never happen was the last thing on their minds.

Sigh. A lot of other people at work were using Twitter, so now I am too. If I join anything else, I’ll need to think of a good way to organize all of my web presences. I guess this blog can be the mothership, and contain links to everything else.

So far I seem to be using Twitter as a dumping ground for my Google Talk status messages, so they are no longer lost to the mists of the intertubes when I switch to a new one. I don’t ever foresee myself updating it on the go from a mobile phone — I just don’t have that much of a desire to remain connected. Being off the grid can be good sometimes.

As you’ve probably figured out by now, the past several months on Cyde Weys Musings have been characterized by my extreme inactivity. That wasn’t unintentional. I just had to make enough of a break from the old direction of the site in order to feel comfortable taking it in a new direction. Apparently, a couple of months’ worth of guilt over writing very little for you all was enough to overcome the threshold of taking this site in a more personal direction.

I’ve finally realized that what I enjoy more than anything else is writing fiction, yet so long as I was writing non-fiction on here quite regularly, that wasn’t going to happen. I might’ve been a bit optimistic when I proclaimed that the secret to getting into the mood to write is by writing, because I clearly wasn’t doing it. I’ve finally turned that corner.

So the new personal direction on here will consist of me writing fiction frequently and publishing all of it. I know that doesn’t sound very personal, especially since I won’t be revealing any sort of intimate details about myself like so many other bloggers, but believe me, it is. It’s a frightening thing putting your writing out there for all to see and critique. While I got over that hump with my non-fiction awhile ago, fiction is still a much more sensitive area. But I’ve finally gotten over it, and I’d rather my work be read and possibly disliked than languish in obscurity on my hard drive. I’m not saying you shouldn’t be critical — but please be kind. Saying someone’s opinion piece sucks is one thing, but saying someone’s fiction sucks is another whole world of authorial insult.

So, keep your eyes peeled. I’m going to keep at this for awhile come hell or high water. I’m not really attempting to become a professional writer or anything (though I wouldn’t mind it if that did end up happening). I’m content to write for a small audience here. That what I write is read by a few people is enough.

For completeness’s sake, here’s a list of all of the works of fiction I’ve published on this site so far that I consider at least passable. I hope to at least double this number of works within a month.

• Invasion Day
• A Stay At Motel Z
• One Humble Cloth
• For Want of a Bolt

“So much for Her Majesty’s elite unit,” Lieutenant Nottingham said softly as he sank down into the canvas netting of the wall-mounted fold-up bench. The electric lantern above his head drifted lazily to and fro.

“It was a great idea, only suffering from a complete lack of execution,” I responded, trying to find some humor in the situation. We continued hearing the gentle pitter-patter of enemy fire against the outside of our behemoth. It sounded oh so distant.

“And to think, all for want of a bolt,” Nottingham continued morosely.

“A bolt, indeed,” I responded. I gave up on tinkering with the steam relay junction and sat across from Nottingham.

“What do we do now? Surrender?” Nottingham asked. He was leaving the decision up to me, even though he out-ranked me. The death of our commanding officer had shattered the chain of command. I could still picture the tortured look on his face as he sank down from the exposed cupola above the hatch, the fatal rifle shot having turned half of his insides to scrapple.

“And give them use of the behemoth? Never!” My resolve strengthened. I knew victory was no longer within our reach, but we could still avoid absolute defeat.

“Perhaps they’d find more use for it than we,” Nottingham said.

“No, we know what we must do,” I snapped, not in the mood for his attempt at gallows humor. The rain of lead against the outside of our hull abated slightly, passing into a more stochastic, syncopated murmur of clinks. Perhaps they were finally realizing our impenetrability. Perhaps they lost the pressing need for urgency upon realizing the complete hopelessness of our circumstances.

“We’ll die in the explosion too, you know,” he said. I could see a hint of fear in his eyes, which he hurriedly hid by slouching forward even more and resting his forehead on his palm. He was not ready.

“We only have one other choice then – set it in motion dead ahead, right into the river,” I resolved. “It won’t be recoverable after that.”

“Fine,” Nottingham hastily agreed. It was better than the alternative.

We set about the startup process with automaton-like efficiency. I slammed open all the intake relays leading to the boilers, material stresses be damned. The behemoth would find finally be unleashed to discover its maximum capabilities. Nottingham triggered the ignition and a loud whistling overtook us for a spell as the immense boilers began building up pressure. Evidently our enemies could hear it too, because the rate of ineffectual fire increased back into more steady volleys. I could sense their fear even through the two-foot steel armor surrounding us on all sides.

We waited as the indicator needles on the engineering panel slowly rotated to the right. I looked at Nottingham but he averted my gaze, too intent on attempting to discern some pattern from the now just-barely-audible reports of enemy fire.

When the majority of the needles passed the first marked threshold, I threw the main handle forward, and the behemoth roared to life. The control room creaked all around us as the first two locomotive legs began lifting up. The floor turned several degrees from straight down. I fought against the disorientation that proven to be the biggest hardship in finding a driver for the behemoth.

The needles continued rising as the automatic loader worked at full bore, delivering fresh lumber to the fireboxes as quickly as space permitted. The movement of the behemoth settled into a regular swaying motion, not unlike being at sea. I counted the movements across ten strides and satisfyingly noted that our speed was increasing at a brisk pace. I no longer heard any bullets pinging off the outside of our vehicle, but I could not discern whether that was because we had outrun our enemies, or simply because our steam engine was so deafeningly loud. As I reached for my earplugs, Nottingham interrupted me.

“One last word for Major Littlepage!” he shouted.

“Fine,” I shouted back, wincing as the mechanical noises around me grew ever louder.

Nottingham attempted to shout out some kind of eulogy or prayer for the late Major, I did not know which. I could not hear him over the roar of our furious machine. Instead, I pondered on his ill-fated mission, and the absurd flaw in the behemoth’s design. If I ever met the idiot that decided to put the spare parts locker on the outside of this rust bucket, I’d have a few choice words for him. Apparently the possibility of there being a need to make minor repairs to the behemoth in combat had escaped the designer’s mind. The courageous Major had died for want of a bolt, shot down as soon as he lifted himself through the hatch. He’d never even made it within twenty feet of the parts locker.

As soon as Nottingham stopped talking, I reached for my earplugs and jammed them in.

I peered through the periscope. The Rhine was still directly ahead of us, exactly as I had remembered. Except that now we were much, much closer to it, and drawing closer with an accelerating pace. Driving the behemoth deep into the river would guarantee its end just as much as detonating it from within, as no machine would ever be able to drag it out again save another behemoth — and after this travesty, there likely wouldn’t be another. Additionally, it would afford us some attempt at escape.

The pitch of the floor changed again as the behemoth entered the downwards sloping flood plain. It wouldn’t be long now. We picked up even more speed, running across the landscape at a rate I hadn’t even thought our behemoth capable of. She saw her fate before her and charged after it with a head full of steam. I glanced down at the boiler indicators, all of which were pegging against their final redlines so hard that the needles were bending. This would be the behemoth’s final blaze of glory.

I motioned to Nottingham, but he already knew what to do. He raced back to the hatch, stepping gingerly over the Major’s body in the process, and prepared to open it. With a tremendous splash, our first leg entered the edge of the river, throwing up a huge plume of spray. The other legs followed suit. Our forward momentum steadily dropped. The indicators sagged. Shortly enough we were in water up to the bottom of the main body. The boilers choked and died as they flooded with water, that infernal whistling giving its last dying gasp. Water began rushing into the cabin from a thousand different spots in the floor. The legs locked up and the behemoth began pitching forward.

“Now!” I yelled at Nottingham at the top of my lungs. He couldn’t hear me over the din, nor was he even looking at me, but he sensed that the time was right nonetheless, and with a giant push, he heaved the hatch upwards and clambered through it. I took one last look at the flooding cabin, sneering in disgust at the discarded steering wheel bobbing in the water, its sheered-off attachment bolt gleaming brightly in the newfound sunlight, and made to follow after him.

Excellent news! My favorite web browser, Mozilla Firefox, has gained market share yet again and now commands 21.53% of the market. That’s a far cry from several years ago when Firefox was just coming out and Internet Explorer was by far the dominant browser. I still remember all of those sites that only worked in Internet Explorer, and because alternative browsers weren’t very popular, companies got away with it. Now that Internet Explorer “only” has 67.55% of the market, no one dares make a site that requires it, thus alienating a whole third of potential customers. I can’t even remember the last time I saw an IE-only site.

Unfortunately, while Firefox’s market share is gaining, the software itself is gaining more and more problems. Firefox crashing has become a daily occurrence for me. I remember when it used to stay alive for months — barely. At least it now saves the list of open tabs and allows you to resume them upon restarting, but you still lose lots of logged in sessions and it’s just a big hassle. And while it is true that most Firefox crashes can be traced back to Adobe’s Flash plugin, there’s no excuse for Firefox allowing a bug in a plugin to crash the whole application. Google Chrome found a fix for this by running each tab as a separate process. Firefox needs to do the same, or else it won’t keep gaining market share for much longer. As much as I hate to admit it, Firefox has some pretty significant flaws.

One night, on the precipice of sleep, a fragment materialized in my mind that I immediately aroused myself to record. It led to the following account:

Why does this cloth billow so? It has seen the sunshine of many days and the darkness of many nights, the twilights of oh so many yesterdays. Yet with each fresh gust it prances with greater ease in the sky. While all else around it grow old and reach their ends, it grows younger, shedding thread after thread each year, losing its resistance to the wind’s capricious ways. It is growing younger into sheer nothingness, the day when the cloth will be no more. Until then, it waves with ever-increasing vigor, eager at the prospect of its end.

How many a battle has this cloth seen? More than any mere cotton, it has served as a rallying point in countless skirmishes, as a standard in actions, as a focal point in parade. It is made of such humble materials, yet it has seen more history than the greatest person that ever lived. Its tapestry of patchwork, correcting uncountable battle wounds, attests to that. Many a man has died because of and on behalf of this cloth, a sad trifle it is fortunate it cannot be troubled by.

Such bold and regular geometric patterns! (Only spoiled by hasty repairs.)

Such vivid dashes of color! (T’would be better had it not faded with age.)

Such a spritely dance through the sky!

What is it in this cloth that spurns men to action? Why have so many cheerfully lept into warfare beneath it? What poisoning effect does it have on otherwise thinking minds that drags them down into brutality, even while proclaiming that it stands for all that is noble and patriotic? Yet the uncaring merry cloth tangos in the clear blue sky, oblivious to all it has wrought and all that it will wreak.

This cloth will go on to see the end of the world — the end of the world, caused on its account.

How dare this cloth billow so?

Today was a long yet exhilarating day. It started at around 7:00am when we woke up and boarded the Metro downtown to catch the inauguration. We had purple tickets allowing us onto the Capitol grounds, but alas, the purple line was mishandled horribly, and we didn’t actually get in. Frustration was running high in the crowd, but when noon came around, the mood quickly improved, and when the oath was said and the cannon fire commenced, a cry of jubilation emerged from the crowd. I’d be lying if I said I wasn’t doing my part with that.

Up next was the inaugural speech, and luckily, my dad brought a portable RadioShack radio. An impromptu group of listeners surrounded us as Obama gave his inauguration speech, eager for any sort of live broadcast in the land-of-no-Jumbotrons. Our small huddled mass, and the others just like it around similar devices, was a microcosm of the Obama movement itself. We had people of all ages and colors: two white teen-aged girls, some middle-aged African-American women, an older ex-hippie couple, and others. At least of the listeners were crying at some point in the speech. After it was over, several people profusely thanked my dad for sharing the experience.

That’s the memory I’m always going to keep from this event. Yes, we didn’t get to use our tickets, but it was amazing all the same. I would not have had the same shared experience sitting at home watching it on the television. Afterward, we headed over to the parade route and stood in the freezing cold until Obama and Biden drove by. Then we departed, navigating the mess of a city completely swamped by its most massive event ever.

There was one particular point in Obama’s speech that really surprised and impressed me. When I heard it, I almost thought I had misheard it, and I looked at my dad for confirmation. He seemed genuinely surprised as well. Yet here it is in the transcript:

We will not apologize for our way of life, nor will we waver in its defense, and for those who seek to advance their aims by inducing terror and slaughtering innocents, we say to you now that our spirit is stronger and cannot be broken; you cannot outlast us, and we will defeat you.

For we know that our patchwork heritage is a strength, not a weakness. We are a nation of Christians and Muslims, Jews and Hindus – and non-believers. We are shaped by every language and culture, drawn from every end of this Earth; and because we have tasted the bitter swill of civil war and segregation, and emerged from that dark chapter stronger and more united, we cannot help but believe that the old hatreds shall someday pass; that the lines of tribe shall soon dissolve; that as the world grows smaller, our common humanity shall reveal itself; and that America must play its role in ushering in a new era of peace.

How far we’ve come in these two short decades since George Bush Sr. uttered this infamous statement: “No, I don’t know that atheists should be considered as citizens, nor should they be considered patriots. This is one nation under God.”

Now that’s change we non-believers can believe in.

Recently, some Republicans thought it would be a brilliant idea to distribute a CD to members of the Republican National Convention containing a song titled “Barack the Magic Negro”. The general response was about as predictable as the sunset, and rightly consisted of outright condemnation over such disgusting and overt racism. But get this: the excuse of the Republican who distributed the song was that it was a parody.

Really? A song sung to the same tune as the pop hit “Puff the Magic Dragon” is parody? Anyone with two neurons to rub together can figure that out. Yes, “Barack the Magic Negro” is parody. Racist parody. Racism and parody are not mutually exclusive, so asserting that it’s parody isn’t a defense against the actual charge. The rebuttal thus rings completely hollow. The idiot should’ve just apologized instead of offering this pathetic attempt at an excuse, thus digging the hole even deeper.

If the Republican Party wants to recover from the doldrums they’re currently languishing in, they might want to stop being openly racist. Most people don’t like that. Just saying.