Jon Callas | 17 Aug 08:04 2013

Reply to Zooko (in Markdown)


Also at http://silentcircle.wordpress.com/2013/08/17/reply-to-zooko/

# Reply to Zooko

(My friend and colleague, [Zooko
Wilcox-O'Hearn](https://leastauthority.com/blog/author/zooko-wilcox-ohearn.html) wrote an
open letter to me and Phil [on his blog at
LeastAuthority.com](https://leastauthority.com/blog/open_letter_silent_circle.html).
Despite this appearing on Silent Circle's blog, I am speaking mostly for myself, only slightly for Silent
Circle, and not at all for Phil.)

Zooko,

Thank you for writing and your kind words. Thank you even more for being a customer. We're a startup and
without customers, we'll be out of business. I think that everyone who believes in privacy should support
with their pocketbook every privacy-friendly service they can afford to. It means a lot to me that you're
voting with your pocketbook for my service.

Congratulations on your new release of [LeastAuthority's S4](https://leastauthority.com) and
[Tahoe-LAFS](https://tahoe-lafs.org/trac/tahoe-lafs). Just as you are a fan of my work, I am an
admirer of your work on Tahoe-LAFS and consider it one of the best security innovations on the planet.

I understand your concerns, and share them. One of the highest priority tasks that we're working on is to get
our source releases better organized so that they can effectively be built from [what we have on
GitHub](https://github.com/SilentCircle/). It's suboptimal now. Getting the source releases is
harder than one might think. We're a startup and are pulled in many directions. We're overworked and
understaffed. Even in the old days at PGP, producing effective source releases took years of effort to get
down pat. It often took us four to six weeks to get the sources out even when delivering one or two releases
per year.

The world of app development makes this harder. We're trying to streamline our processes so that we can get a
release out about every six weeks. We're not there, either.

However, even when we have source code to be an automated part of our software releases, I'm afraid you're
going to be disappointed about how verifiable they are. 

It's very hard, even with controlled releases, to get an exact byte-for-byte recompile of an app. Some
compilers make this impossible because they randomize the branch prediction and other parts of code
generation. Even when the compiler isn't making it literally impossible, without an exact copy of the
exact tool chain with the same linkers, libraries, and system, the code won't be byte-for-byte the same.
Worst of all, smart development shops use the *oldest* possible tool chain, not the newest one because
tool sets are designed for forwards-compatibility (apps built with old tools run on the newest OS) rather
than backwards-compatibility (apps built with the new tools run on older OSes). Code reliability almost
requires using tool chains that are trailing-edge.

The problems run even deeper than the raw practicality. Twenty-nine years ago this month, in the August
1984 issue of "Communications of the ACM" (Vol. 27, No. 8) Ken Thompson's famous Turing Award lecture,
"Reflections on Trusting Trust" was published. You can find a facsimile of the magazine article at
<https://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf> and a text-searchable
copy on Thompson's own site, <http://cm.bell-labs.com/who/ken/trust.html>.

For those unfamiliar with the Turing Award, it is the most prestigious award a computer scientist can win,
sometimes called the "Nobel Prize" of computing. The site for the award is at <http://amturing.acm.org>.

In Thompson's lecture, he describes a hack that he and Dennis Ritchie did in a version of UNIX in which they
created a backdoor to UNIX login that allowed them to get access to any UNIX system. They also created a
self-replicating program that would compile their backdoor into new versions of UNIX portably. Quite
possibly, their hack existed in the wild until UNIX was recoded from the ground up with BSD and GCC.

In his summation, Thompson says:

    The moral is obvious. You can't trust code that you did not totally
    create yourself. (Especially code from companies that employ people
    like me.) No amount of source-level verification or scrutiny will
    protect you from using untrusted code. In demonstrating the
    possibility of this kind of attack, I picked on the C compiler. I
    could have picked on any program-handling program such as an
    assembler, a loader, or even hardware microcode. As the level of
    program gets lower, these bugs will be harder and harder to detect.
    A well installed microcode bug will be almost impossible to detect.

Thompson's words reach out across three decades of computer science, and yet they echo Descartes from
three centuries prior to Thompson. In Descartes's 1641 "Meditations," he proposes the thought
experiment of an "evil demon" who deceives us by simulating the universe, our senses, and perhaps even
mathematics itself. In his meditation, Descartes decides that the one thing that he knows is that he,
himself, exists, and the evil demon cannot deceive him about his own existence. This is where the famous
saying, "*I think, therefore I am*" (*Cogito ergo sum* in Latin) comes from.

(There are useful Descartes links at: <http://www.anselm.edu/homepage/dbanach/dcarg.htm> and
<http://en.wikipedia.org/wiki/Evil_demon> and <http://en.wikipedia.org/wiki/Brain_in_a_vat>.)

When discussing thorny security problems, I often avoid security ratholes by pointing out Descartes by
way of Futurama and saying, "I can't prove I'm not a head in a jar, but it's a useful assumption that I'm not." 

Descartes's conundrum even finds its way into modern physics. It is presently a debatable, yet legitimate
theory that our entire universe is a software simulation of a universe . Martin Savage of University of
Washington <http://www.phys.washington.edu/~savage/> has an interesting paper from last November
on ArXiV <http://arxiv.org/pdf/1210.1847v2.pdf>.

You can find an amusing video at 
<http://www.huffingtonpost.com/2012/12/24/universe-computer-simulation_n_2339109.html> in
which Savage even opines that our descendants are simulating us to understand where they came from. I
suppose this means we should be nice to our kids because they might have root.

Savage tries to devise an experiment to show that you're actually in a simulation, and as a mathematical
logician I think he's ignoring things like math. The problem is isomorphic to writing code that can detect
it's on a virtual machine. If the virtual machine isn't trying to evade, then it's certainly possible (if
not probable -- after all, the simulators might want us to figure out that we're in a simulation). Unless,
of course, they don't, in which case we're back not only to Descartes, but Godel's two Incompleteness
Theorems and their cousin, The Halting Problem.

While I'm at it, I highly, highly recommend Scott Aaronson's new book, "Quantum Computing Since
Democritus" <http://www.scottaaronson.com/democritus/> which I believe is so important a book that I
bought the Dead Tree Edition of it. ([Jenny Lawson](http://thebloggess.com) has already autographed
my Kindle.)

Popping the stack back to security, the bottom line is that you're asking for something very, very hard and
asking for a solution to an old philosophical problem as well as suggesting I should prove Godel wrong. I'm
flattered by the confidence in my abilities, but I believe you're asking for the impossible. Or perhaps
I'm programmed to think that.

This limitation doesn't apply to just *my* code. It applies to *your* code, and it applies to all of us.
(Tahoe's architecture makes it amazingly resilient, but it's not immune.) It isn't just mind-blowing
philosophy mixed up with Ken Thompson's Greatest Hack.

Whenever we run an app, we're trusting it. We're also trusting the operating system that it runs on, the
random number generator, the entropy sources, and so on. You're trusting the CPU and its microcode.
You're trusting the bootloader, be it EFI or whatever as well as
[SMM](http://en.wikipedia.org/wiki/System_Management_Mode) on Intel processors -- which could
have completely undetectable code running, doing things that are scarily like Descartes's evil demon.
The platform-level threats are so broad that I could bore people for another paragraph or two just
enumerating them.

You're perhaps trusting really daft things like [modders who slow down entropy
gathering](http://hackaday.com/2013/01/04/is-entropy-slowing-down-your-android-device/)
and [outright
bugs](http://android-developers.blogspot.com/2013/08/some-securerandom-thoughts.html). 

Ironically, the attack vector you suggest (a hacked application) is one of the harder ways for an attacker
to feed you bad code. On mobile devices, apps are digitally signed and delivered by app stores. Those app
stores have a vetting process that makes *targeted* code delivery hard. Yes, someone could hack us, hack
Google or Apple, or all of us, but it's very, very hard to deliver bad code to a *specific* person through
this vector, and even harder if you want to do it undetectably.

In contrast, targeted malware is easy to deploy. Exploits are sold openly in exploit markets, and can be
bundled up in targeted advertising. Moreover, this *has* happened, and is known to be a mechanism that's
been used by the FBI, German Federal Police, the Countries Starting With the Letter 'I' (as a friend puts
it), and everyone's favorite The People's Liberation Army. During Arab Spring, a now-defunct
government just procured some Javascript malware and dropped it in some browsers to send them passwords
on non-SSL sites.

Thus, I think that while your concern does remind me to polish up my source code deployment, if we assume an
attacker like a state actor that targets people and systems, there are smarter ways for them to act.

I spend a lot of time thinking, "*If I were them, what would I do?*" If you think about what's possible, you
spend too much time on low-probability events. Give yourself that thought experiment. Ask yourself what
you'd do if you were the PLA, or NSA, or a country starting with an 'I.' Give yourself a budget in several
orders of magnitude. A grand, ten grand, a hundred grand, a million bucks. What would you do to hack
yourself? What would you do to hack your users without hacking you? That's what I think about.

Over the years, I've become a radical on usability. I believe that usability is all. It's easy to forget it
now, but PGP was a triumph because you didn't have to be a cryptographer, you only had to be a techie. We
progressed PGP so that you could be non-technical and get by, and then we created PGP Universal which was
designed to allow complete ease of use with a trusted staff. That trusted staff was the fly in the ointment
of Silent Mail and the crux of why we shut it down -- we created it because of usability concerns and killed it
because of security concerns. Things that were okay ideas in May 2013 were suddenly not good ideas in
August. I'm sure you've noted when using our service our belief in usability. Without usability that is
similar to the non-secure equivalent, we are nothing b
 ecause the users will just not be secure.

I also stress Silent Circle is a *service*, not an app. This is hard to remember and even we are not as good at it
as we need to be. The service is there to provide its users with a secure analogue of the phone and texting
apps they're used to. The difference is that instead of having utterly no security, they have a very high
degree of it.

Moreover, our design is such to minimize the trust you need to place in us. Our network includes ourselves as
a threat, which is unusual. You're one of the very few other people who do something similar. We have
technology and policy that makes an attack on *us* to be unattractive to the adversary. You will soon see
some improvements to the service that improve our resistance to traffic analysis.

The flip side of that, however, is that it means that the device is the most attractive attack point. We can't
help but trust the OS (from RNG to sandbox), bootloader, hardware, etc.

Improvements in our transparently (like code releases) compete with tight resources for improvements in
the service and apps. My decisions in deploying those resources reflect my bias that I'd rather have an A
grade in the service with a B grade in code releases than an A in code releases and a B service. Yes, it makes it
harder for you and others, but I have to look at myself in the mirror and my emphasis is on service quality
first, reporting just after that. Over time, we'll get better. We've not yet been running for a year.
Continuous improvement works.

I'm going to sum up with the subtitle of the ACM article of Ken Thompson's speech. It's not on his site, but it
is on the facsimile article:

    To what extent should one trust a statement that a program is free
    of Trojan horses? Perhaps it is more important to trust the people
    who wrote the software.

Thank you very much for your trust in us, the people. Earning and deserving your trust is something we do
every day.

Regards,
Jon

Gmane