Jan Kundrát | 22 Feb 15:05 2012

Message Composing and SMTP

Hi Mildred,
this is going to be a slightly larger reply, so I hope you don't mind me
writing that to the list as well.

First of all, some bits about the MIME message composer and the SMTP
client code:

> I am wondering if we should fix the current code, or start with
> something completely new. Possibly using an external library. SMTP
> doesn't seems so hard to implement but when something goes wrong (error
> from the server) the behaviour is quite undefined.
> If we can use a small external library to deal with that, I'd prefer. I
> don't think of any reason to reinvent the wheel there.
> For now, I did fix SMTP, just barely. I also made it possible to
> view/edit tags on the messages. I think the next thing to do would to
> fix the message editor, and then go on improving other things.
> I have a question, are you open to external dependencies ?
> I don't think you have code that implements RFC2822 (message format).
> I'd like to grab some external code for that as I'd prefer spending time
> on the interface.

The existing message submission is, as you've already seen, in pretty
bad shape. The message composer was written by me, and it's essentially
a quick & dirty code to hack together a text/plain message, set a few
crucial headers and pass the blob into the message submission agent (MSA).

The MSA si implemented either through a connection to the SMTP server
over TCP, or simply as a sendmail-compatible process. The SMTP client is
based on Witold Wysota's QwwSmtpClient [1] with a few bugfixes. The
QwwSmtpClient is not picked by any distributions, so I decided to simply
ship it as part of Trojita.

Right now and in the immediate future, this approach is "enough" and
would fly even when someone extends the message composer to include
"support for attachments" -- and when it's done, I'd really want the
composer to support full-blown MIME messages with all whistles, ie.
support for forwarding a complex message as an attachment etc. If I was
writing this code, I'd probably work with a tree-like representation of
the MIME message and then serialize it to the MIME format (with the
RFC2822 encapsulation) when asked to send. The communication between the
Composer and the SMTP client would be limited to actually just "here's a
complete message and here's its envelope, please send it".

In slightly more distant future, though, I'd like to support advanced
IMAP/SMTP features like BURL. Suppose a situation when I'm on my cell
phone and just have received a mail with some huge attachment. The mail
also contains a few lines of text clarifying what's it all about. The
message therefore has the following structure:

- multipart/related (the whole message body)
  +-- text/plain (a short text with the description)
  +-- application/octet-stream (a binary attachment, 15MB)

Please note that Trojita allows me to have a look at the original
message and downloading just the small text/plain part, not the huge

I'm on a vacation and I want to pass the message to my colleague who's
supposed to handle my duties while I'm gone. In most of the current
MUAs, I'd have to download both the short text message and the huge
attachment, somehow process it, add my short note and submit the
resulting blob of data to the SMTP server (and upload yet another copy
of the data via IMAP to my sent folder). This is how the resulting copy
would look like:

- multipart/related
  +-- text/plain with *my* short note asking my colleague for help
  +-- message/rfc822 with all headers of the original message
    +-- multipart/related (body of the original message_
      +-- text/plain with the *original* note
      +-- application/octet-stream, the huge original attachment

If my IMAP server supported CATENATE extension, I could instead build
this new message in my Drafts folder in a way which needs transmitting
just the new text/plain with my short note and asking the IMAP server to
use the existing data for the rest, without having to upload stuff again.

Now we have a new message on the IMAP server, but we got to send it.
Right now, the only supported way of doing so is through an SMTP
extension where instead of a regular DATA command followed by the full
message, you ask it to retrieve the actual data from the IMAP server.
Things are more complex in real world, one can mix the uploaded data
with something to be downloaded from some URL etc etc.

The catch is that this needs support in all of the IMAP server (some
already do), SMTP server (some already do) and the MUA (Trojita could do
that in future). Even in the MUA, one has to be careful, as this special
mode has to be supported by both the message composer (which suddenly
doesn't have to build the full MIME source of a message and pass it to
the MSA) and by the MSA itself (which has to know when to issue a DATA
and when to use BURL to ask the SMTP server to download a part of the
message itself).

This mail is pretty long already, sorry for that, but I believe that
it's important for you to know these details so that we can design a
good solution which won't need rewriting in future. Now the question is,
how to do that best in Trojita.

I don't care about the QwwSmtpClient -- if you'd like to get rid of that
and replace it with somewhere, please be my guest. Anything which is
GPL-compliant will do. The second option is fixing QwwSmtpClient --
again, I'm fine with that as well.

One possible contender is the vmime library [2]. I haven't used that
before, but its API is not Qt-based. That's not a huge problem, but if
there's a reasonable alternative written in Qt, I'd prefer that.

Which leads me to the Qt Messaging Framework [3]. It's Nokia's own
attempt at working with e-mails. Its SMTP code already supports (or at
least looks like it supports) the BURL extension, which is great. On the
other hand, the code of the SMTP stuff is rather closely integrated with
the rest of the framework -- for example there are callbacks like:


or their own logging features etc etc. In Trojita, we do not have that
(and I haven't considered providing a QMailStore-compatible API -- that
might work or might not, don't know). The question is how to use this in
Trojita in a reasonable manner. Things which I'd like to avoid are:

- carrying a massive patchlist against their version
- forking the code altogether

So one way of working with that is providing at least some stub
interfaces of the code they call. Another one is moving this integration
stuff into some well-isolated places in their code (and persuading them
to integrate these features). I'm not sure they're going to agree with
that, it'd be a rather intrusive change for them and would not get them
any benefits, IMHO. Someone will have to write that code as well (and
that someone would be either you or me). I'll ask them what's their
stance on this anyway.

About the composer -- I think that a tree-like interface with custom
roles (as already implemented by Imap::Mailbox::Model in Trojita) is the
way to go here. The composer shall use some underlying model (which
would have to support modifications, of course), and it'd be interesting
if this model can be "backed up" by data coming fmro various places at
once, maybe an attachment from some IMAP connection and also some
locally-saved modifications. In short-term, the whole functionality
doesn't really have to be implemented, what I'd like to have is a
working interface for now. Something like this:

class ComposedMessageModel: public QAbstractItemModel {
// regular MVC methods like index(), parent() and data() here

/**  <at> short Remove the specified message part and its children from the
message */
void removePart(const QModelIndex &);

/**  <at> short Append another part to the message

The parent is either an invalid QModelIndex to refer to the root part,
or a valid one to refer to an existing multipart/* message part to serve
as a parent for the new one.

The rawData are the real data to be stored in the MIME structure. Any
MIME encoding and similar transformations are preformed by this class
automatically, ie. the data shall not be encoded.

FIXME: add some headers etc etc
void insertPart(const QModelIndex &parent, const QByteArray &mimeType,
const QByteArray &rawData);

/**  <at> short Same as above, but for retrieving data from another
QModelIndex (which shall belong to another model, like the IMAP one etc) */
void insertPart(const QModelIndex &parent, const QModelIndex &part);

/**  <at> short Build full message source for submission to an SMTP client */
QByteArray toMimeData() const;

This model will be used by the message Composer as its "backing store"
holding the message source. When the COmposer is done and the message is
about to be sent, the model gets passed to the MSA. The MSA already
knows whether the method being used supports BURL or not (maybe because
it's just talking to a sendmail-compatible process). If it doesn't
support BURL, it just calls the toMimeData() (or rather calls something
like this:

	model->data(QModelIndex(), RoleMessageFullMimeSource);

ie. asking the model for full MIME source for the root index. If the MSA
wants to use BURL, it can walk the tree of indexes and find out if the
particular part is accessible through some URL or not.

The QMF uses their own abstraction here, not the QModelIndex-based API.
I'd say that the model-view architecture is actually superior.

What's your opinion on this?

> Currently, I'm using git more like a way to synchronize the code base
> between different computers. I didn't put any effort into rebasing to
> get a clean patch set. I'm open to comments and I am ready to
> merge/rebase from the trunk frequently. And if I see something that
> could be committed as a whole, I won't hesitate to ask for a pull request.

Cool. The thing is that I'm a lazy person, so I tend to work directly on
master, without bothering with feature branches etc -- especially given
that I was essentially just a single developer. It would therefore be
rather odd to insist on other contributors working "properly". Anyway,
I'll try to use branches more thoroughly.

> What I am working on currently, is the minimum to be able to use Trojita
> as a mail client. It involves:
> - fixing SMTP
> - fixing message creation
> - implementing tags

I forgot about one more thing -- in order to "play well" on the
Internet, Trojita shall really include the In-Reply-To and References
headers when replying to some message. I'll have to extend the IMAP code
to ask for these items.

> I'll probably wait a little bit until I ask for pull requests. I also
> don't know what are your standards about the UI. If you look at what
> I've done, the UI is in a very poor state. But I am not breaking
> anything already there so ... would it qualify to integrate the trunnk?

Well if you look at the "UI" I have made... I'm not a UI designer, so my
criteria are highly subjective. If it isn't ugly, it can get in easily.


[1] http://blog.wysota.eu.org/index.php/2009/05/13/qwwsmtpclient-released/
[2] http://www.vmime.org/


Trojita, a fast e-mail client -- http://trojita.flaska.net/