Peter Holditch

I've seen several posts in the public WebLogic server transactions newsgroup in which people have had problems with transactions spread across multiple servers.

The gist of these problems is always that they have two EJB components in two different servers. Bean One on Server One executes in the context of a JTA transaction in which it calls Bean Two on Server Two, thus propagating the transaction across to the second server. Finally, the first method ends, and the transaction commits.

When the time to commit comes, the transaction is aborted. Accompanying the abort is a cryptic exception, javax.transaction.HeuristicMixedException. Since this comes up repeatedly, I thought I would dedicate this month's column to unravelling what's going on here.

First the euphemism: in the world of transactions, transaction managers don't like to sound too pessimistic, so if they think the world is coming to an end, they don't engage in unnecessary hyperbole, as it might upset the system administrators. Instead, they just quietly say that a heuristic outcome has occurred. That means that your carefully orchestrated ACID transaction has just ended with all the data potentially inconsistent, and you'd better go check your databases. Or, to put it another way, the world is at an end!

So with that explanation out of the way, let's think about what's happening here.

The call to Bean 1 has started a transaction that is associated with the thread of control and will propagate through to any database updates and the like that it performs. Then, it calls Bean 2. This will flow a request over to Server 2, where the second bean will be executed. Somehow, the transaction is being flowed over, accompanying the RMI request. Usually, server-to-server communication between WLS instances is done using the T3 protocol, and one of the things that can be achieved using T3 is the attaching of information about transactions to a request. So under the covers, the two WebLogic instances have exchanged notes about the transaction, and now both can participate, and at a later date commit or abort it as a single unit.

Allow Me to Digress...

At this point, a small digression is in order. The latest version of WLS, 7.0, is fully J2EE 1.3-certified, which mandates, among other things, support for EJB 2.0. This in turn mandates application server interoperability using RMI/IIOP. The IIOP protocol, like T3, allows context information such as active transactions to be passed along with method invocations. So, with WLS 7.0, this same scenario could be played out with one of the servers being a WLS instance and the other a different app server (although I don't know why you'd want one - but I'm biased!). Anyway, back to the plot...

Something I haven't mentioned so far in all of this is security. The client that initiated this request connected to Server 1 at some point using an InitialContext with a username and password. When the method on Bean 1 is invoked, the EJB container will check that the logged-in user is allowed to execute the method. Likewise, the EJB container on Server 2 will do the same check before it allows access to the method on Bean 2. This implies that the security ID associated with the request has also flowed over the wire, along with the transaction ID, and indeed it has. This is another capability of the T3 protocol. Incidentally, it's also another requirement of EJB 2.0 container interoperability - a protocol called CSIv2 is used to flow a user identifier over an IIOP connection and this too is supported by WLS 7.0.

The request has flowed from Server 1 to Server 2, along with the transaction and the caller's user ID, and it gets executed there and the result is returned. So far, so good. Now Bean 1 returns to the client and, if the transaction was container-managed, then the container will call commit. It is the job of the transaction co-coordinator (part of the JTA subsystem in Server 1 in this case) to synchronize the prepare and commit phases of this transaction. It will see that there is a participant on Server 2 and send a message to the server instructing it to perform the prepare phase of the two-phase commit protocol. It's at this point that the problems arise. This call isn't being done on behalf of the client, rather it is initiated by the JTA subsystem. What security ID will flow across the wire? In fact, it will be the WebLogic "system" user on whose behalf the prepare phase is executed. It would be a pretty serious security flaw if any Tom, Dick, or Harry could randomly prepare or commit transactions, so the use of the system ID closes this potential security hole. However, we now need to take a look at the security model underpinning WLS 6 (this area is another that has been substantially upgraded in WLS 7). In version 6, two servers will trust each other if they share the same system password. There is no greater degree of trust than trusting the system user, so for the prepare (and subsequent commit) phases to be allowed, the two servers must share a system password.

And Back to the News...
In conclusion, every time this has arisen on the newsgroup (remember, that's where this all started) the solution has been to make sure that all WLS instances that participate in a transaction share a common system password. Also note that transactions can happen that you're not aware of; for instance, if you send a message to a remote JMS queue, a transaction may well be started on your behalf by the JMS subsystem. Remember, next time you see a HeuristicMixedException, check your passwords!

Resources

• WebLogic server transactions newsgroups: