Features Download
From: martin odersky <martin.odersky-p8DiymsW2f8 <at> public.gmane.org>
Subject: Binary compatibility: status and outlook
Newsgroups: gmane.comp.lang.scala.user
Date: Friday 29th April 2011 17:32:54 UTC (over 5 years ago)
When 2.8 was around the corner there was a big discussion about (the lack
of) binary compatibility in Scala.

My answer then was that we would keep binary compatibility for minor
revisions and look into ways how to address the problem for major
The first half of the promise was kept. 2.8.1 was binary compatible with
2.8. Now that 2.9 is around the corner, the question is what can we do for
binary compatibility now and in the future?

If you are too impatient to read the details: we not there yet, but are
making progress. In particular, we have developed the basic technology that
lets us maintain binary compatibility for future releases.

Now to the details: There are in fact many sources of binary compatibility.
Some of the most common ones are:

1. New methods or fields are added to existing traits. Implementations (but
not clients) of these traits need to be recompiled.

For instance, the new collection libraries add methods `par' and `seq' to
all collection traits, as well as several other methods.

2. Classes and companion objects are moved to different packages, with type
aliases and forwarders in a package object in the old location. Any clients
of those classes have to be recompiled.

For instance, the 2.9 standard library has moved several annotation classes
from package `scala' to package `scala.annotation'.

3. Lazy vals are added to a class. This shifts the indices
used to access bitmaps in all subclasses.

4. Methods get different signatures. Parameter types might be widened, or
result types narrowed or widened. Again, all callers of such a method need
to be recompiled.

For instance, collection operations such as `zip' used to take an Iterable
as argument, now they take a GenIterable (GenIterable is the common
superclass of Iterable and ParIterable). There are many other examples like

What can we do to find and fix these? We have been working on a tool which
will detect binary compatilities between different versions of a library.
The tool is in prototype stage now. We will make it available as soon as it
is in a usable state. With the tool, we can have a better idea where binary
incompatibilities have been generated, and can roll back if they are

Some binary incompatibilities are not accidental, but the result of
conscious generalizations and enrichments of the libraries. We do not want
to sacrifice these possible improvements for binary compatibility. What we
do instead is search for technological solutions. For problems of the first
three kinds (new trait members, moved classes, new lazy vals), we are
working on a tool that can fix these issues by rewriting code on the client
side. For problems of the third kind (changed method signatures), we have
developed a new solution based on bridge methods. Let's say you have a

  def combine(other: Seq[String]): Unit

and you want to generalize the method to take Traversables instead:

  def combine(other: Traversable[String]): Unit

That change is source compatible (all code calling the old `combine' will
work with the new one), but it is not binary compatible, because the new
method handle has type Traversable in its signature where the old one had
Seq. Since callers refer to a method by giving its name and full signature,
callers compiled against the old method will not work with the new one.

To fix this kind of problem (which can be diagnosed using our tool), we can
add a bridge method:

@bridge def combine(other: Seq[String]): Unit =
  combine(other: Traversable[String])

Since there is now a method with the right signature, binary compatibility
is maintained. At the same time, the bridge method will be invisible to
freshly compiled Scala code, so all recompiled code will pick up only the
new method, not the bridge. This also ensures that bridges do not affect
overloading or overriding relationships for the other methods; they are
purely there for bytecode generation. Using bridge methods, we can
compensate the significant changes caused by the arrival of parallel
collections in 2.9.

So these are the schemes we are working on. We are reasonably confident
they will solve Scala's binary compatibility once they are ready. For 2.9,
bridge methods ensure that quite a lot of code compiled against 2.8 will
continue to operate, but my no means all code. So it is very much advised
recompile your projects for 2.9. Recompilation should in most cases be
painless because 2.9 is by-and-large source compatible with 2.8. There's
exception: If your application uses features that were already deprecated
2.8, it might find these features removed in 2.9. So it's a good idea to
rid of deprecation warnings before upgrading.

  -- Martin

Martin Odersky
Prof., EPFL and CEO, Scala Solutions
PSED, 1015 Lausanne, Switzerland
CD: 2ms