Monday, December 8, 2008

Searching for a needle in the haystack "Los Angeles"

In late October, I visited Los Angeles, CA - the reason being an information technology conference (namely Microsoft PDC - quite a good one by the way). Besides attending the conference, there are several other things you can do in Hollywood's home. Of course, there's the whole Hollywood experience - the Walk of Fame amongst other things. There's LA's neighborhoods, e.g. the glamorous Bel Air, which is certainly interesting to drive through. As a PDC attendee, you get to see the well-known Universal Studios, no explanation needed. Leaving all that aside, the city itself is not what you would expect - or, more precisely, what I had expected before my first visit in 2005. Streets are mostly empty - deserted in comparison other cities, you would think that people don't live in the fast lane in the undisputed capitol of America's film industry. I'm thinking New York City (been there many times; without a doubt a favorite of mine), the ever pulsating and forward-moving city life - cars in the busy streets, sidewalks filled with stressed people making their ways through Manhattan's grid of avenues. No, not in LA. If you invert my thought, you could think of LA as a relaxing place to be - the predominantly mild weather encountered throughout California contributes favorably.

One other thing you can do while you're in "The Angels" is searching for a needle in the haystack of buildings and streets on a sunny morning (28°C) in fall - the needle being a filming location of a Showtime TV show I enjoy watching for it's originality in terms of story and characters - Dexter. If you've heard of or seen it already, you know it's set in Miami Beach, FL. Unsurprisingly, that usually means filming was not done in that part of the US. Before the trip, I had done some research and ultimately found a reference, which confirmed the theory and actually covered three locations in LA, all of which should be known to an audience familiar with season one. The hockey stadium a victim of the so-called "ice truck killer" was found in. The hospital where "Tony Tucci", a surviving victim of the ice truck killer, was found. The third was in close proximity to my hotel, an easy choice: one of many sites where the ice truck killer recreated a memorable moment of Dexter's youth, in this instance him standing beside a palm with a black smiley face on it and a junkyard in the background.

The site provided the address, I fed my iPhone with the data via wifi and thereby preloaded the maps (unfortunately, there's no way to explicitly do so for all levels of detail). Off I was to follow the 3km route before the conference started, a piece of cake for the most part. A worthwhile feature though would be automatic map rotation based on the current / last GPS vector, that would make navigation more intuitive. Having arrived at the supposed filming location, I compared the surroundings with a screenshot of the scene in question - Season 1, Episode 4, "Let's Give the Boy a Hand", 39:42 excluding opening credits...

...and, I can confirm that the site matches the scene location, it (still) looks more or less identical.

The major missing piece is the palm tree that featured the smiley face. I certainly didn't expect to find it there, it was a requisite (identical to the one in the photo). The tree was placed inside a circle-shaped traffic refuge. That's still there, palm-less though.

I wish I had taken a picture of me reenacting the scene :)

Monday, June 23, 2008

On libcurl, OpenSSL, and thread-safety

The cURL project with its libcurl is a frequent choice of developers requiring a feature-rich HTTP client. Indeed, libcurl is a good choice, it supports HTTP/1.1, several authentication mechanisms (including Kerberos / SPNEGO authentication), and HTTPS, to name a few important aspects. It can also be used in multi-threaded applications - well - it can but you have to be aware of some fundamental facts in order to avoid random segmentation faults or aborts - Linux is assumed here...

First, to avoid pitfall #1, be sure to disable the use of signals of the library by adding the following line of code to the initialization phase of your application:
curl_easy_setopt(handle, CURLOPT_NOSIGNAL, TRUE);

This option deactivates code that works around the fact that DNS lookups initiated via gethostbyname cannot be interrupted, there's no timeout facility. The libcurl developers of course knew about the one generic way to interrupt the current thread of execution and execute code in that context - signals.

In order to interrupt gethostbyname, libcurl saves the current state of execution via setjmp, initializes a timeout of n seconds by calling alarm, which results in signal delivery at that point-in-time. If gethostbyaddr does not return in time, the signal handler associated with SIGALARM, the delivered signal, is being called. The handler restores the original state using longjmp to rewind execution and indicates a timeout via the return value.
longjmp(state, 1);

signal(SIGALARM, signalhandler);

if (setjmp(state) == 0) {
/* snip */
else {
/* timeout */

In a single-threaded application, that works fine because alarm only affects the process and the main thread is identical to the process itself. In a multi-threaded application, however, the fact that the process gets to handle the signal and not a particular thread, the workaround can fail because an unrelated thread could handle SIGALARM. While this strategy may work in the case of LinuxThreads, it does not with NPTL, the state-of-the-art Linux thread library that implements POSIX semantics. The relevant part of the specification is as follows:

"There were two possible choices for alarm generation in multi-threaded applications: generation for the calling thread or generation for the process. The first option would not have been particularly useful since the alarm state is maintained on a per-process basis and the alarm that is established by the last invocation of alarm() is the only one that would be active.

Furthermore, allowing generation of an asynchronous signal for a thread would have introduced an exception to the overall signal model. This requires a compelling reason in order to be justified."

So, by setting the cURL option, you disable DNS timeouts but thereby avoid related segmentation faults in multi-threaded applications. If you don't plan to invest more time into fixing this issue, that's an acceptable solution.

Pitfall #2 is merely related to libcurl, but it does cause random "crashes" if you intend to use libcurl along with HTTPS requests in multi-threaded applications. The module of interest is OpenSSL, the primary backend for libcurl and HTTPS. By default, using libcurl and HTTPS in multiple threads can lead to crashes, even if you do not share libcurl-specific handles or, more generally, memory. The reason for that is OpenSSL, which is not thread-safe by default.

Rather than providing a thread-safe library out-of-the-box, the OpenSSL team decided to leave this as an exercise for the (documentation) reader and / or user. OpenSSL provides callbacks that define functions for serializing access to resources. Two sets of callbacks exist, one provides access to a static set of locks that can be locked and unlocked. The other allows for allocation, deallocation and lock / unlock of a lock object.
By default, these are not implemented. In other words, code using OpenSSL with threads can fail unless the developer read the relevant parts of the documentation and implemented the required callbacks for all supported platforms correctly.
Hmmm. Don't get me wrong, I'm the first to vote for reading the documentation / specification before writing code and I'm a big fan of generic and flexible interfaces but, IMHO, the OpenSSL team took the easy path here. What I'd have done is implement locking for supported platforms right in the library to cover all direct and indirect (e.g. libcurl) users.

So, should libcurl define these callbacks (it doesn't)? That's a tough one. The main issue is that these callback are process-global and thus must be implemented by every single library or module that makes use of OpenSSL. Depending on the linking scenario of these and OpenSSL (dynamic or static), there's no truly correct implementation strategy for these callbacks - modules could overwrite each others callbacks and cause memory leaks, hangs, and segmentation faults as a result of an implementation mismatch at an arbitrary point-in-time.

Consider the following scenario: a process loads library A which depends on libcurl. Consequently, library A implemented the callbacks and assigns these at load time.
A library used by library A, THIRDPARTY, uses libpq, the official PostgreSQL driver. The driver also uses OpenSSL and initializes the callbacks in the function PQinitSSL.
Now, in case of the following sequence of events, this scenario can cause a hang because the unlock implementation does not match the lock implementation:

1) Initialization of library A
2) Library A initializes OpenSSL locking callbacks
3) Library A receives libcurl request from multiple threads, that result in repeated lock and unlock callback invocations
4) Thread 1 invokes the lock callback in OpenSSL
5) A new request handled by thread 2 results in the initialization of THIRDPARTY, which loads libpq and calls PQinitSSL. The initializes overrides the locking callbacks
6) Thread 1 invokes the unlock callback which has no effect on the previously locked lock object because the implementation changed between lock and unlock. Application hang or undefined behavior

Of course there are other scenarios that can lead to problems, all of them caused by the global locking callbacks that must be implemented for thread-safe operation. My point is that whenever global resources and multiple libraries are involved, chances are that these cannot coexist.

I recommend one of the following two approaches to following the OpenSSL contract while also minimizing collisions with other libraries or modules:

Solution A) Beware of other libraries in your locking callback implementation

Implement and install your callback knowing that other libraries might have installed callbacks already. In particular, do not install callbacks if callbacks are already in place (and consistent) to avoid causing hangs or undefined behavior. Uninstall callbacks on unload to avoid crashes on subsequent callback invocations. Uninstall the callbacks only if these represent callbacks installed by your library - previously installed callbacks must not be affected. Additionally, new callbacks installed by other modules after the initialization of your library should not be affected as well (it's better to leave them in place rather than having no callbacks installed).

The following pseudo-code implements these recommendations:
if (!all_callbacks_are_installed) {
else {
/* Do not interfere with existing callbacks. */

/* Uninstall callbacks to avoid segmentation faults after unload. */
/* Only uninstall callbacks owned by this library. */
if (installed_callbacks == library_callbacks) {
uninstall callbacks();

Solution B) Use a private OpenSSL library (less desirable)

If dependencies allow, link OpenSSL statically so that callbacks are not shared and conflicts can be avoided. The major drawback here is that OpenSSL cannot be updated independent from your implementation which is critical in the case security updates must be applied. Used libraries depending on OpenSSL must be linked statically as well, which might not be an option in the case of proprietary libraries or libraries not available as an archive. If you go down that road, make sure not to export OpenSSL symbols (GCC: compile with -f visibility=hidden) to prevent other libraries from accessing your private (and possibly incompatible) copy. This does have its issues, but sometimes, there's no other way.

Whatever the reasoning against implementing the locking code directly in the library was, it unnecessarily complicates the task of writing stable multi-threaded code for developers. Fortunately, if some thought goes into the callback implementation, the facility provided by OpenSSL is good enough for completing the task of enabling the thread-safe operation of OpenSSL and libcurl as well as other OpenSSL dependencies.

With these two issues addressed, libcurl should integrate just fine in your muti-threaded code.

Sunday, May 25, 2008

Home, Sweet New Home

Throughout the last 7 years or so, I've been living in a small apartment approx. 40 minutes by feet from downtown Linz. Several months ago, I was fortunate enough to find a more spacious replacement right in downtown Linz, only minutes away from (almost) everything you can think of. Additionally, it's situated next to a gas station that's open for business on a 24/7 basis, a rare comfort in European countries. So, should my supply of Red Bull (I can't help myself) come to an end at 4:46am in the course of a programming spree, I can resort to my backup next door :)

After two weeks of moving essentials and most other things, it's finally done. I'm positive that my new home will give me a a general boost, definitely so. It might also help me to establish a routine morning run before work, a nice 6 km waterside course only meters away is waiting to be challenged...

The living room is not ready yet. Nonetheless, being a technology enthusiast, indispensable subtle technology references are already in place ;)

P.S.: The UNIX license plate can be obtained from The Open Group online store. I once owned a Digital UNIX / Compaq Tru64 UNIX plate, an adaption of the original. At that time I was an Alpha / Tru64 advocate and user. As always, nothing can replace the original...

Saturday, May 3, 2008

Cisco VPN Client on Leopard 10.5.2

Just a quick note for those of you struggling with Cisco's VPN client on Leopard, Mac OS X 10.5.2. My initial attempt was based on client version The result was the following error message indicating that no active network interfaces are available:

Error 51: Unable to communicate with the VPN subsystem.
Please make sure that you have at least one network interface that is currently active and has an IP address and start this application again.

Needless to say, that was a false statement. So I updated to, that didn't fix the issue. What fixed it was a minor upgrade to build 100, version, the VPN connection has become fully functional from there.

The essence is: With Leopard 10.5.2, use

How do you share your runs? Do you?

I don't. My current solution for logging runs is a Windows-based software called SportsTracks. It supports the basic set of per-workout metadata as well as input from devices such as a heart rate monitor or GPS receiver. I use Garmin's Forerunner 305 to record workaround duration, distance, heart rate and course, the import works like a charm. Additionally, SportsTracks provides you with an extensive toolkit for analyzing individual workouts by means of statistics and interactive charts, and generating reports covering multiple workouts. What I love about workouts is the logging ritual. Logging a new run after a hard workout and browsing through the data that was silently recorded throughout the last minutes or more likely hours, that's something I really enjoy. I'm obsessed with my distance totals too ;) That left aside this facility serves as kind of a virtual trainer that can actually provide objective feedback...

I have two problems with my current solution. One, SportsTracks is Windows-based and will probably never run on Linux or Mac OS X. My current system runs on Mac OS X, so I have to fall back to a Windows system to import my new workouts and analyze my progress. SportsTracks used .NET, thus, the chances of a Linux or Mac OS X version are practically non-existent. I don't want to go the virtual machine route, not for one application.

Two, and more importantly, there's no easy way to share all data of my past runs via the internet. Yes, I know - not true. What you can do is export maps and workout data as HTML, an automatic upload via FTP can be triggered as well. Right, my idea of sharing my runs though is to provide visitors with an experience that is similar to SportsTracks. For example, with every new run, I can easily check new distance totals in SportsTracks. In the case of per-workout exports, these totals are not available. Another aspect is interactivity with tables and charts, static images are of course limited to a single view of data.

I guess, I'm looking for some kind of service that provides runners with an easy way of sharing workout data and visitors with a clean yet feature-rich interface for browsing through the shared data. Nike+ provides a nice interface that goes into the right direction. Understandably, it only supports Nike+ input (which is why I had to manually migrate run-by-run from Nike+ to SportsTracks when I switched to the Forerunner :/ ). It fails in respect to sharing though, there's no way I know of to share all runs as opposed to a single selected run.

So, my fellow runners: How do you share your runs? Do you share them at all?

Sunday, April 27, 2008

A first: 26.2 km

I'm not yet sure about my next race goal - half-marathon or marathon - but I still decided to try a new distance which can be helpful in either case (thinking overdistance training). Before yesterday, my longest run had been the half-marathon distance, around 21 km. After the first 11 km of my run, I committed to a distance of 26 km, which required me to turn around after the initial 13 km (I toyed with a goal of 28 km, didn't want to push too hard though).

The first 21 km had been relatively easy, the last 5 km however were very difficult and continually prompted me to give up and be satisfied with the half-marathon distance. The energy was still there, my legs were killing me though, I already saw cramps coming (although that never happened). After 26.2 km and around 3 hours of non-stop running (or jogging), I finished the workaround and managed to reach the goal of completing 26 km, quite satisfying indeed :)

This workout provided me with a foretaste of what to expect throughout a marathon and definitely made me appreciate the challenge of 1) finishing a marathon and 2) doing so in a reasonable time, more than ever before. I knew it wouldn't be easy, but quite frankly I didn't expect 5 km more to become much of a problem.

My previous training strategy was to run new distances for as long as they represent a major obstacle, so I'll do more 26 km workouts throughout the next weeks until I can safely say that I master the distance.
Thereafter, the next milestone might be the dreaded 32 km mark - where a marathon is said to begin and work against the runner. We'll see about that :)

Sunday, April 13, 2008

OMV Linz Marathon 2008

I've been running for more than a year now, with around 1,100 completed kilometers. I don't know why I had decided to start running in the first place, what I do know is that this was one of the best decisions of my life. Being a stressed-out software engineer, I enjoy the relaxation and energy boost running brings along, it's also the simplest (run anytime, anywhere) and least expensive workout possible.

My racing debut was the "Linzer Sparkassen Citylauf 2007", a half-marathon. In contrast to today, I was less prepared for the race as well as the pace required to complete it in a satisfying time, the result was 02:16:46. I completed the first 10k in around 57 minutes, however, I was forced to slow down during the second half because my energy depot was practically empty. 

Today, 6 month after my HM debut, I participated in the "OMV Linz Marathon 2008", my competition of choice was once again the half-marathon. I was unsure about participation because f a foot injury I contracted one week ago during training. During a 5k test run yesterday, the injury was fine during the beginning, but was clearly noticeable in the end. Today morning, one hour before the race, I was in doubt about participating as my foot got worse. Nevertheless, I decided to pull through!

At 9:15 am, the raced started on a freeway bridge with around 14,000 participants. My intention was to finish before the two-hour mark, my training was based on a plan from Herbert Steffny for 1:59:00. To ensure a steady and adequate pace, I was using the Garmin Forerunner 305, which not only records the course via GPS, it also provides an instant pace average that is accurate enough for relying on it during the race.

The first 19k were comparably easy, way easier than last time as well as during training runs and way faster than ever before. I completed the 10k in around 57 minutes and still had a chance to easily finish within the first 2 hours. Between 19k and 21k, I increased my pace to make sure I wouldn't miss my goal. The "hammer-man" tried to intervene at that point, however, I managed to maintain the pace (around 5:00) while heading towards the finish line. It's true, the cheering audience does help quite a lot. I passed the finish line with an official net time of 01:57:35, slightly better than expected and with relatively little effort during the first 19k. I was very satisfied with the result, particularly considering my injury, which could have completely thwarted my plans!

Paricipating in a race amongst 14,000 fellow runners was a fantastic experience, finishing in 01:57:35 was excellent and exceeded my expectations for the day. In contrast to last year, I took carbo-loading seriously this time. On a retrospective note, I think the full depot of carbo-hydrates enabled my effortless run until the 19k mark, it clearly showed.

It's always interesting to analyze the race by means of the data recorded by Forerunner. My average pace was 05:30. While I did not plan for it, I managed a negative split - the second half page average was slightly higher than the first. Here's a chart covering pace and heart rate during the race:

The following image shows the course based on GPS data:

The OMV Linz Marathon was excellent, I thoroughly enjoyed every second (well, almost ;) ) and will take part again next year unless another, more interesting event turns up.

Only one open question remains for the next event:
new half-marathon goal OR marathon?

Thursday, April 10, 2008

Don't throw your Macbook Air out just yet, let there be silence

I've been waiting for my personal Macbook Air for around two weeks and could not wait to get hold of that machine extraordinaire. I expected to receive a notebook that is well-manufactured and manages to combine a minimalistic and elegant design with a solid level of performance.
I loved my new main computer - my Air - from the very beginning (yes, m-a-i-n computer, I know I'm crazy ;) ). Unfortunately, my enthusiasm faded after 1 hour of working with the default OS X installation. A Macbook Air is a beautiful machine, but I also expected it to be beautifully silent during "normal use". Well, much to my regret, it was not! Whenever the CPU load increased moderately (e.g. due to watching a YouTube video) and lasted for longer periods of time, the Air increased the fan speed to the factory maximum of 6200 rpm. At 6200, the buzz of the Air's fan can be incredibly annoying, particularly if you're sensitive to fan noise. At 2500 rpm, the fan is practically unnoticeable - the Air's pretty aggressive about cooling though, so, these low levels are rare in the non-idle-case.

My research lead me to believe that this NOT an issue affecting my Air in particular, it's behavior by design. Lots of Air owners seem to be annoyed by the fan noise, none of them have found satisfying workarounds and some even returned their Airs or got them replaced in hope for a "working" unit.

Rather than returning the Air or requesting a replacement to fix the issue, I investigated further and tried various ways to bring silence to the otherwise flawless and fascinating Air experience. I experimented with different energy saving configurations, software for controlling the fan rpm minimum as well as CPU voltage and frequency and also changed OS X kernel module configurations to alter the fan behavior. While all of these had noticeable effects on the situation, none of them prevented the fan controller from increasing revolutions to the maximum of 6200.
The fan is controlled by the so-called System Management Controller (SMC). Software allows you to modify the rpm minimum, that however could not solve the issue. Apple now provides an SMC firmware update that affects fan control. According to the release notes, this update could potentially worsen the situation because it apparently tries to lower CPU temperature further to avoid hangs and unexpected shutdown (of a single core and/or the whole system). I tried the update, it didn't have any noticeable effect in either direction.

Despite the 6200 rpm targeted by SMC, CPU temperature levels remain in a perfectly safe region (60 - 70 °C) at all times. So, knowing that the rpm increase is unjustified in respect to temperature levels, I changed the rpm minimum to 4500 and 6200. In respect to CPU temperature at 100% CPU utilization, the temperature diff between 4500 (maximum acceptable level that can be considered comfortable) and 6200 rpm is pretty much insignificant. Knowing that, my goal was to redefine the rpm maximum to fully address the issue at stake. I was unable to find any way to do that during the first round of research. Most utilities, including SMC Fan Control, merely allow you to change the minimum. As it turns out though, that software package provides a console application smc that can be used to changed all writable SMC parameters, not just the rpm minimum. With the help of the source of a Linux kernel driver controlling the fan via SMC, I identified the key representing the SMC parameter controlling the rpm maximum and gave the new maximum a try by executing smc as follows:

smc -k F0Mx -w $(python -c 'print hex(4500 << 2)[2:]')

Additionally, I reset the minimum to 1500 to conserve the fan at CPU idle levels:

smc -k F0Mn -w $(python -c 'print hex(1500 << 2)[2:]')

These commands only change minimum and maximum, revolutions still remain in control of SMC. The results have been fantastic, the Air has been quiet and cool ever since enforcing the new limits. Execute these commands in a startup item to make changes permanent (for the sleep / wakeup case, use SleepWatcher).

Now, I can finally say that it is a real pleasure to work with this elegant and capable notebook, I enjoy every minute of doing so. I strongly encourage you to give this workaround a try before throwing the Air out (at your own risk), you definitely miss something!