Features Download
From: Jon Callas <jon-1c5t7ZzLY0Ednm+yROfE0A <at> public.gmane.org>
Subject: Reply to Zooko (in Markdown)
Newsgroups: gmane.comp.security.cryptography.randombit
Date: Saturday 17th August 2013 06:04:38 UTC (over 4 years ago)
Hash: SHA1

Also at http://silentcircle.wordpress.com/2013/08/17/reply-to-zooko/

# Reply to Zooko

(My friend and colleague, [Zooko Wilcox-O'Hearn](https://leastauthority.com/blog/author/zooko-wilcox-ohearn.html)
wrote an open letter to me and Phil [on his blog at LeastAuthority.com](https://leastauthority.com/blog/open_letter_silent_circle.html).
Despite this appearing on Silent Circle's blog, I am speaking mostly for
myself, only slightly for Silent Circle, and not at all for Phil.)


Thank you for writing and your kind words. Thank you even more for being a
customer. We're a startup and without customers, we'll be out of business.
I think that everyone who believes in privacy should support with their
pocketbook every privacy-friendly service they can afford to. It means a
lot to me that you're voting with your pocketbook for my service.

Congratulations on your new release of [LeastAuthority's S4](https://leastauthority.com) and
Just as you are a fan of my work, I am an admirer of your work on
Tahoe-LAFS and consider it one of the best security innovations on the

I understand your concerns, and share them. One of the highest priority
tasks that we're working on is to get our source releases better organized
so that they can effectively be built from [what we have on GitHub](https://github.com/SilentCircle/).
It's suboptimal now. Getting the source releases is harder than one might
think. We're a startup and are pulled in many directions. We're overworked
and understaffed. Even in the old days at PGP, producing effective source
releases took years of effort to get down pat. It often took us four to six
weeks to get the sources out even when delivering one or two releases per

The world of app development makes this harder. We're trying to streamline
our processes so that we can get a release out about every six weeks. We're
not there, either.

However, even when we have source code to be an automated part of our
software releases, I'm afraid you're going to be disappointed about how
verifiable they are. 

It's very hard, even with controlled releases, to get an exact
byte-for-byte recompile of an app. Some compilers make this impossible
because they randomize the branch prediction and other parts of code
generation. Even when the compiler isn't making it literally impossible,
without an exact copy of the exact tool chain with the same linkers,
libraries, and system, the code won't be byte-for-byte the same. Worst of
all, smart development shops use the *oldest* possible tool chain, not the
newest one because tool sets are designed for forwards-compatibility (apps
built with old tools run on the newest OS) rather than
backwards-compatibility (apps built with the new tools run on older OSes).
Code reliability almost requires using tool chains that are trailing-edge.

The problems run even deeper than the raw practicality. Twenty-nine years
ago this month, in the August 1984 issue of "Communications of the ACM"
(Vol. 27, No. 8) Ken Thompson's famous Turing Award lecture, "Reflections
on Trusting Trust" was published. You can find a facsimile of the magazine
article at <https://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf>
and a text-searchable copy on Thompson's own site, <http://cm.bell-labs.com/who/ken/trust.html>.

For those unfamiliar with the Turing Award, it is the most prestigious
award a computer scientist can win, sometimes called the "Nobel Prize" of
computing. The site for the award is at <http://amturing.acm.org>.

In Thompson's lecture, he describes a hack that he and Dennis Ritchie did
in a version of UNIX in which they created a backdoor to UNIX login that
allowed them to get access to any UNIX system. They also created a
self-replicating program that would compile their backdoor into new
versions of UNIX portably. Quite possibly, their hack existed in the wild
until UNIX was recoded from the ground up with BSD and GCC.

In his summation, Thompson says:

    The moral is obvious. You can't trust code that you did not totally
    create yourself. (Especially code from companies that employ people
    like me.) No amount of source-level verification or scrutiny will
    protect you from using untrusted code. In demonstrating the
    possibility of this kind of attack, I picked on the C compiler. I
    could have picked on any program-handling program such as an
    assembler, a loader, or even hardware microcode. As the level of
    program gets lower, these bugs will be harder and harder to detect.
    A well installed microcode bug will be almost impossible to detect.

Thompson's words reach out across three decades of computer science, and
yet they echo Descartes from three centuries prior to Thompson. In
Descartes's 1641 "Meditations," he proposes the thought experiment of an
"evil demon" who deceives us by simulating the universe, our senses, and
perhaps even mathematics itself. In his meditation, Descartes decides that
the one thing that he knows is that he, himself, exists, and the evil demon
cannot deceive him about his own existence. This is where the famous
saying, "*I think, therefore I am*" (*Cogito ergo sum* in Latin) comes

(There are useful Descartes links at: <http://www.anselm.edu/homepage/dbanach/dcarg.htm>
and <http://en.wikipedia.org/wiki/Evil_demon>
and <http://en.wikipedia.org/wiki/Brain_in_a_vat>.)

When discussing thorny security problems, I often avoid security ratholes
by pointing out Descartes by way of Futurama and saying, "I can't prove I'm
not a head in a jar, but it's a useful assumption that I'm not." 

Descartes's conundrum even finds its way into modern physics. It is
presently a debatable, yet legitimate theory that our entire universe is a
software simulation of a universe . Martin Savage of University of
Washington <http://www.phys.washington.edu/~savage/>
has an interesting paper from last November on ArXiV <http://arxiv.org/pdf/1210.1847v2.pdf>.

You can find an amusing video at  <http://www.huffingtonpost.com/2012/12/24/universe-computer-simulation_n_2339109.html>
in which Savage even opines that our descendants are simulating us to
understand where they came from. I suppose this means we should be nice to
our kids because they might have root.

Savage tries to devise an experiment to show that you're actually in a
simulation, and as a mathematical logician I think he's ignoring things
like math. The problem is isomorphic to writing code that can detect it's
on a virtual machine. If the virtual machine isn't trying to evade, then
it's certainly possible (if not probable -- after all, the simulators might
want us to figure out that we're in a simulation). Unless, of course, they
don't, in which case we're back not only to Descartes, but Godel's two
Incompleteness Theorems and their cousin, The Halting Problem.

While I'm at it, I highly, highly recommend Scott Aaronson's new book,
"Quantum Computing Since Democritus" <http://www.scottaaronson.com/democritus/>
which I believe is so important a book that I bought the Dead Tree Edition
of it. ([Jenny Lawson](http://thebloggess.com) has already
autographed my Kindle.)

Popping the stack back to security, the bottom line is that you're asking
for something very, very hard and asking for a solution to an old
philosophical problem as well as suggesting I should prove Godel wrong. I'm
flattered by the confidence in my abilities, but I believe you're asking
for the impossible. Or perhaps I'm programmed to think that.

This limitation doesn't apply to just *my* code. It applies to *your* code,
and it applies to all of us. (Tahoe's architecture makes it amazingly
resilient, but it's not immune.) It isn't just mind-blowing philosophy
mixed up with Ken Thompson's Greatest Hack.

Whenever we run an app, we're trusting it. We're also trusting the
operating system that it runs on, the random number generator, the entropy
sources, and so on. You're trusting the CPU and its microcode. You're
trusting the bootloader, be it EFI or whatever as well as [SMM](http://en.wikipedia.org/wiki/System_Management_Mode)
on Intel processors -- which could have completely undetectable code
running, doing things that are scarily like Descartes's evil demon. The
platform-level threats are so broad that I could bore people for another
paragraph or two just enumerating them.

You're perhaps trusting really daft things like [modders who slow down
entropy gathering](http://hackaday.com/2013/01/04/is-entropy-slowing-down-your-android-device/)
and [outright bugs](http://android-developers.blogspot.com/2013/08/some-securerandom-thoughts.html).

Ironically, the attack vector you suggest (a hacked application) is one of
the harder ways for an attacker to feed you bad code. On mobile devices,
apps are digitally signed and delivered by app stores. Those app stores
have a vetting process that makes *targeted* code delivery hard. Yes,
someone could hack us, hack Google or Apple, or all of us, but it's very,
very hard to deliver bad code to a *specific* person through this vector,
and even harder if you want to do it undetectably.

In contrast, targeted malware is easy to deploy. Exploits are sold openly
in exploit markets, and can be bundled up in targeted advertising.
Moreover, this *has* happened, and is known to be a mechanism that's been
used by the FBI, German Federal Police, the Countries Starting With the
Letter 'I' (as a friend puts it), and everyone's favorite The People's
Liberation Army. During Arab Spring, a now-defunct government just procured
some Javascript malware and dropped it in some browsers to send them
passwords on non-SSL sites.

Thus, I think that while your concern does remind me to polish up my source
code deployment, if we assume an attacker like a state actor that targets
people and systems, there are smarter ways for them to act.

I spend a lot of time thinking, "*If I were them, what would I do?*" If you
think about what's possible, you spend too much time on low-probability
events. Give yourself that thought experiment. Ask yourself what you'd do
if you were the PLA, or NSA, or a country starting with an 'I.' Give
yourself a budget in several orders of magnitude. A grand, ten grand, a
hundred grand, a million bucks. What would you do to hack yourself? What
would you do to hack your users without hacking you? That's what I think

Over the years, I've become a radical on usability. I believe that
usability is all. It's easy to forget it now, but PGP was a triumph because
you didn't have to be a cryptographer, you only had to be a techie. We
progressed PGP so that you could be non-technical and get by, and then we
created PGP Universal which was designed to allow complete ease of use with
a trusted staff. That trusted staff was the fly in the ointment of Silent
Mail and the crux of why we shut it down -- we created it because of
usability concerns and killed it because of security concerns. Things that
were okay ideas in May 2013 were suddenly not good ideas in August. I'm
sure you've noted when using our service our belief in usability. Without
usability that is similar to the non-secure equivalent, we are nothing b
 ecause the users will just not be secure.

I also stress Silent Circle is a *service*, not an app. This is hard to
remember and even we are not as good at it as we need to be. The service is
there to provide its users with a secure analogue of the phone and texting
apps they're used to. The difference is that instead of having utterly no
security, they have a very high degree of it.

Moreover, our design is such to minimize the trust you need to place in us.
Our network includes ourselves as a threat, which is unusual. You're one of
the very few other people who do something similar. We have technology and
policy that makes an attack on *us* to be unattractive to the adversary.
You will soon see some improvements to the service that improve our
resistance to traffic analysis.

The flip side of that, however, is that it means that the device is the
most attractive attack point. We can't help but trust the OS (from RNG to
sandbox), bootloader, hardware, etc.

Improvements in our transparently (like code releases) compete with tight
resources for improvements in the service and apps. My decisions in
deploying those resources reflect my bias that I'd rather have an A grade
in the service with a B grade in code releases than an A in code releases
and a B service. Yes, it makes it harder for you and others, but I have to
look at myself in the mirror and my emphasis is on service quality first,
reporting just after that. Over time, we'll get better. We've not yet been
running for a year. Continuous improvement works.

I'm going to sum up with the subtitle of the ACM article of Ken Thompson's
speech. It's not on his site, but it is on the facsimile article:

    To what extent should one trust a statement that a program is free
    of Trojan horses? Perhaps it is more important to trust the people
    who wrote the software.

Thank you very much for your trust in us, the people. Earning and deserving
your trust is something we do every day.

Version: PGP Universal 3.2.0 (Build 1672)
Charset: us-ascii

CD: 3ms