Monday, May 14, 2007

Getting Subversion (neon) to work with Kerberos on SLES10

Just in case you were wondering why Subversion fails to authenticate against an Apache-based, Kerberos-enabled Subversion repository on SuSE Linux Enterprise Server 10...

$ svn info https://host/path/to/repository
svn: PROPFIND request failed on '/path/to/repository'
svn: PROPFIND of '/path/to/repository': authorization failed (https://host)

For one, SuSE packages a build of neon, the HTTP / WebDAV library the Subversion client uses, that was not configured with Kerberos support:

$ ldd /usr/lib64/libneon.so.24.0.7
libssl.so.0.9.8 => /usr/lib64/libssl.so.0.9.8 (0x00002b5f0548b000)
libcrypto.so.0.9.8 => /usr/lib64/libcrypto.so.0.9.8 (0x00002b5f055d1000)
libz.so.1 => /lib64/libz.so.1 (0x00002b5f05834000)
libexpat.so.1 => /usr/lib64/libexpat.so.1 (0x00002b5f05949000)
libc.so.6 => /lib64/libc.so.6 (0x00002b5f05a6c000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002b5f05c9c000)
/lib64/ld-linux-x86-64.so.2 (0x0000555555554000)

So, by installing the neon source RPM, krb5-devel-1.4.3-19.10.3 and rebuilding the RPM using rpmbuild -bb SPECS/krb5.spec, problem #1 can be solved. Wait, the story isn't over yet.

The situation remains unchanged despite a Kerberos-enabled build:

$ svn info https://host/path/to/repository
svn: PROPFIND request failed on '/path/to/repository'
svn: PROPFIND of '/path/to/repository': authorization failed (https://host)

By enabling neon tracing (by modifying the Subversion 'servers' configuration) and adding additional traces, I found out that gss_init_sec_context fails with GSSAPI major and minor:

No context has been established. Validation error

I checked the Kerberos source:

/usr/src/packages/SOURCES/krb5-1.4.3/src/lib/gssapi/krb5/duplicate_name.c:

...
static OM_uint32
mutual_auth(
OM_uint32 *minor_status,
gss_ctx_id_t *context_handle,
gss_name_t target_name,
gss_OID mech_type,
OM_uint32 req_flags,
OM_uint32 time_req,
gss_channel_bindings_t input_chan_bindings,
gss_buffer_t input_token,
gss_OID *actual_mech_type,
gss_buffer_t output_token,
OM_uint32 *ret_flags,
OM_uint32 *time_rec,
krb5_context context)
{
...
/* validate the context handle */
/*SUPPRESS 29*/
if (! kg_validate_ctx_id(*context_handle)) {
*minor_status = (OM_uint32) G_VALIDATE_FAILED;
return(GSS_S_NO_CONTEXT);
}


There's the problem. gss_init_sec_context validates the input context handle, the validation fails. Looking at the neon source, the bug is obvious:

/usr/src/packages/SOURCES/neon-0.24.7/src/ne_auth.c:

static int
gssapi_challenge(auth_session *sess, struct auth_challenge *parms)
{
...
gss_ctx_id_t context;
gss_name_t server_name;
unsigned int major_status, minor_status;
gss_buffer_desc input_token = GSS_C_EMPTY_BUFFER;
gss_buffer_desc output_token = GSS_C_EMPTY_BUFFER;

clean_session(sess);


if (get_gss_name(&server_name, sess))
return -1;

major_status = gss_init_sec_context(&minor_status,
GSS_C_NO_CREDENTIAL,
&context,
server_name,
GSS_C_NO_OID,
0,
GSS_C_INDEFINITE,
GSS_C_NO_CHANNEL_BINDINGS,
&input_token,
NULL,
&output_token,
NULL,
NULL);

The neon developers forgot to initialize the local variable context with GSS_C_NO_CONTEXT, d'oh...
Apply the following patch to address this:

neon-0.24.7-krb5-auth.patch:

--- neon-0.24.7/src/ne_auth.c 2004-07-05 11:45:42.000000000 +0200
+++ neon-0.24.7-dev/src/ne_auth.c 2007-05-11 22:45:28.000000000 +0200
@@ -343,7 +343,7 @@
static int
gssapi_challenge(auth_session *sess, struct auth_challenge *parms)
{
- gss_ctx_id_t context;
+ gss_ctx_id_t context = GSS_C_NO_CONTEXT;
gss_name_t server_name;
unsigned int major_status, minor_status;
gss_buffer_desc input_token = GSS_C_EMPTY_BUFFER;
@@ -924,15 +924,13 @@
success = 0;

#ifdef HAVE_GSSAPI

Quick Summary:

rpm -ihv neon-devel-0.24.7-20.2..rpm
rpm -ihv krb5-devel-1.4.3-19.2..rpm
Apply the aforementioned patch using patch -p1

It's official: nobody's using Kerberos. OK, just the SLES QA guys ain't... ;)

Nobody's in need of a re-entrant Kerberos 5, why?

IMHO, the Kerberos protocol is the solution for implementing Single-Sign-On (SSO), at least in the context of corporate intranets. Kerberos is practically the only solution available for SSO in a heterogenous computing environment comprised of different operating systems including UNIX variants and Windows. The only time a user ever has to provide a password is during the initial logon on a workstation. The credentials obtained can be passed to specific services and, more importantly, Kerberos allows delegation so that user credentials can be passed from the workstation via service A to service B, for example. Sure, several other SSO implementations exist but, to my knowledge, none of then can offer of the level of transparency and platform-coverage Kerberos can.

There's a catch however, Kerberos 5 implementations available for Linux / UNIX are not re-entrant and thus cannot be used in multi-threaded applications without serializing all code paths accessing the Kerberos API. The main Kerberos implementation I primarily refer to here is MIT Kerberos, the original implementation, Heimdal and friends shouldn't make any difference here however when it comes to a thread-safe implementation. Thinking about it, I haven't reviewed the Java GSSAPI source yet but my guess is that calls are serialized in the JNI part of the library to make sure that Java clients can safely use it across threads.

It might be hard to believe but, for once, Microsoft has managed to create something that is superior to Linux / UNIX equivalents (to be fair, there are other exceptions as well): The Microsoft Windows Kerberos implementation is fully re-entrant. Doesn't come as a surprise to me because Microsoft simply couldn't afford to provide an API that isn't - at last, Windows itself and practically all applications and services are inherently multi-threaded. In my opinion, that is exactly the reason why there's no re-entrant MIT Kerberos available on Linux / UNIX: Except for most GUI applications, common daemons are single-threaded and use a process model instead that often is fork(2) driven:
  • sendmail
  • pop3d / imapd variants
  • sshd
  • ntpd
  • Samba
  • PostgreSQL
  • Apache 1.3 and earlier
  • ...
So, in most cases, the lack of a re-entrant Kerberos implementation is not at all an issue. As a result, the current demand for it doesn't cause that issue to land on the project's TODO list. New implementations and more modern service implementations like Apache 2.0 and later (MPM = {worker|event}) and MySQL are fully multi-threaded, and, once Kerberos support comes into play, would have a problem or at least would need more time to implement Kerberos support (because of the locking business).

Granted, the number of installations dealing with multi-threaded applications (a small subset already) as well as Kerberos (in most cases, it's only an issue for medium to large-scale installations) is not exactly substantial - not substantial enough for supporting the refactoring of existing implementations towards re-entrance. This is a strong case of a minority being discriminated against, please spread the word to make a difference ;)

Seriously, in the Linux / UNIX world, there's currently no satisfactory solution available because even those saying that serialization solves it all are mistaken. Consider a large-scale Kerberos-enabled application using Apache as an interface to application users. In each application module, Kerberos may be used and calls must be synchronized. However, that's easier said than done because that synchronization must include all modules, it must span the whole process to solve this issue. If modules originate from more than one project or vendor, you've lost already - process-wide synchronization would require an agreement across all modules on the means of synchronization. If all modules are controlled by a single project or vendor and/or the source is available for those that are controlled by a different group, re-entrance can be established using a consistent locking protocol (e.g. by providing access to a single, process-wide mutex that is owned by a module all other modules have access to). Otherwise, there's practically no way to work around the lack of re-entrance (except for maybe preloading a library implementing Kerberos symbols that basically wraps all externals and serializes access or rolling your own release based on an open source Kerberos implementation).

You see, this dilemma is nerve-wracking because there's no satisfactory solution. Most likely, several more years will have to pass (where more multi-threaded daemons can emerge and require multi-threaded Kerberos implementations) until this problem will be solved with the availability of a re-entrant implementation...

How do you cope with Kerberos and thread-safety on Linux / UNIX platforms?

Sunday, May 13, 2007

Panting for air on the 4th floor - not anymore

When you go to work, walk up the stairs to the 4th floor and end up panting for air, then, at the latest, you know that the workout comprised of writing lines of code throughout the day - I'm a software developer - is not enough, much to my surprise :) Ever since day 1 of my existence, I've never gone in for sports, the exception being mandatory sports classes during high school, which hardly count. April 2nd, I started to run out of the blue, I simply felt like running - run Horst, run...

Of course, I couldn't have done it without some sort of interesting equipment. For one, I am running in Nike Free's. I bought my first pair around a year ago and the experience was mind-blowing, at least for me. It is said to be like walking or running barefoot. Certainly, it isn't but it comes close. That is what makes the Free a perfect (though expensive) shoe for everyday use, that's what I've been using it before April 2nd. When running, the special properties are particularly beneficial.
Second, I am using my iPod (including Nike+iPod) to record my runs (pace, distance, duration, ...) and keep me entertained. The first aspect is very convenient because the training log writes itself and always contains relatively accurate information on the individual runs (for example, I don't have to estimate the distance myself using Google Maps). More on that later...

So far, I haven't stopped, I'm trying to gradually increase distance and frequency. Since April 2nd, I've been running 155.73 km (you know, that crazy unit equally crazy Europeans are known to think in). It's quite easy to keep going, even for a software developer, because these workouts tend to become addictive (just like hacking the Linux Kernel, but that's a different story ;) ).

As for "panting for air on the 4th floor", well, it's getting much easier already but it's only been 1 1/2 months so far, so there's loads of room for improvement...