Website/LibreOffice Mail Attachment Service

TDF LibreOffice Document Liberation Project Community Blogs Weblate Nextcloud Redmine Ask LibreOffice Donate

Idea

Summary

Enable sending attachments to mailing lists and keep in mind: security, traffic, ease-of-use, costs and misuse potential.

The service is currently being tested.

Rationale

For most FLOSS projects, mailing lists are still the most important communication channel for all the contributors. But, most of the mailing lists used within LibreOffice don't allow attachments to be sent to these lists. Instead, the attachment gets stripped from the original email, and this email gets sent to the mailing list and the mail archives.

The reasons for removing email attachments:

Sending unwanted data to all the subscribed persons shall be avoided
The required amount of data has to be minimized (e.g. because of expensive mobile access, or mediocre Internet connection in some countries)

But removing email attachments also causes issues: Especially new users are unaware of this behavior, thus ...

It requires time to identify an attachment has been stripped
It requires effort to explain people how people can provide the data anyway (e.g. via private mail, uploading to the wiki, providing the data via an external file hosting service)
It doesn't allow to use the common functionality to add attachments via the email clients of the users

Additionally, also experienced community members might benefit if adding attachments would be possible.

If you want to know more, then have a look at the User Stories.

Solution Proposal

If a mail from a subscribed person is sent to a mailing list, and this mail contains an attachment, then the attached file gets extracted from the email and saved on a server. Then, the system embeds a link to this file (or a website containing this file). After this, the mail gets delivered as usual to the mailing lists and the mail archives.

Receiving a Mail

Here is a draft how this may look like for the mailing list members, once they get a mail that originally contained an attachment:

Sending a Mail

Here is a (rather non-technical) draft overview how this may be handled via the system:

Requirements and Constraints

User Stories

The following user stories shall help to understand the functionality from the user's point-of-view. It might be more helpful than reading the technical description.

ID	User Story
Support Mail	Timothy is a new user of LibreOffice and questions concerning its use arise. Also new to the idea of mailing lists, he asks a question there and quickly adds a document to his mail. Without adding the file to the wiki or other document sharing services, the people on the mailing list are able to handle this request immediately. They can click on the link in the request mail and download the document. This even works for people accessing the mailing lists via mail archives.
Low Traffic	Sean is on a trip and handles his LibreOffice mails via his mobile – having only a low bandwidth connection. He doesn't need to worry about people sending large documents to the mailing list, since he can decide what document to open via the document links embedded in the mails. It is helpful for him, that the links are accompanied by some information helping him to decide.
Personal Data	Without intention, Daniel send a document to the mailing lists with some private information. Other people make him aware of that – he asks for deletion of that document. Luckily, the administrators of the attachment server can delete the data which is not “saved for eternity” on the many mailing list archives. From now on, people clicking on the document link in his mail are informed via a web page that this file is no longer available.
Spam	The Bad Company subscribes to known mailing lists and sends spam containing advertisement pictures and manipulated documents. To protect the mailing list members, the LibreOffice mailing list system scans the attachments for known risks and avoid to make these files available to the public.

Requirements

todo

Constraints

todo

Frequently Asked Questions

Question	Answer
What is the main advantage in comparison to storing the files in the Wiki or via other file hoster platforms?	The main advantage is a natural way of handling attachments via the mail programs of the sender. People are sometimes not even aware that (like today) attachments are stripped form the mails - so many people are requested to upload data to the wiki, or to create an account on file hosting platform. But, the quality of the Wiki content gets worse, if more and more (quickly outdated) proposals are stored there. And it is cumbersome to upload the file manually and to manually add links in the mail. This is similar to other file hosting platforms - free services add advertisements and force waiting times when attempting to download the files. Furthermore, files get deleted automatically after a time without "us" having control about that.
How to make sure that the sender agrees to store the data?	In general, sending mails to a mailing list is an implicit agreement to provide the content (body text and attachment) to the public. So it's not different to the today's mail handling. But of course, it might be helpful or even mandatory to e.g. inform the user when he subscribes to the mailing list, or to provide a special link for the user to delete his own content. A great advantage in comparison to the mail itself is, that the data is only stored once on a self-hosted location. So removing the content and preventing further access is quite easy.
How long shall the document files be preserved for public access?	The time the file will be available might depend on the mailing list use - a user mailing list might only require such content for a few days, whereas the design team might require to access this content much longer.
What if people or teams don't want to have their files deleted automatically?	Information on mailing lists are usually meant for short term use - valuable content should always be preserved in a more official place like the Wiki. Thus, we might not provide "unlimited storing capabilities", but we can help the senders to make them aware of that. Examples: Adding the "availability time" to the link that gets embedded into the mail. Sending an automatically generated message to the original sender some days before the file gets deleted by the system.
What if a file is deleted; won't there be many dead links in the mail archives?	That may happen ... but the situation is no worse than using external file hosting platforms. Instead, we should be able to replace the file by a simple web site offering some meta data. For example, we might explain what file had been stored - type, size and name. So the important point is that people are not confronted to "General Error Pages".
What if people send illegal content to the mailing lists?	Although I'm not an expert (=lawyer), I'd say there is no way to totally prevent this - may it be the Wiki, or the mailing list system. Thus, the community, moderators or administrators take care of such content once it has been noticed. Examples: Administrators can be informed to delete the content on request Moderators are informed to manually request "publishing" attachments if people send such mails Moderators are just informed that somebody sent a mail with attachment If a number (e.g. 5) community members report this file as problematic, then the content will get unpublished automatically What we can do anyway, is to scan such attachments for know threats, or to only allow common file types (similar approach in the wiki).
How can we prevent people using our mailing list as a "file hosting service"?	Everybody would be aware of that, because they would have to send a mail with attachment. Thus, it is unlikely that people do that - or the files get quickly deleted by the administrators.
How to prevent that our attachment storing service doesn't run out of space?	That's easy - just provide sufficient space :-) To be serious, it might be required to automatically delete attachments, to restrict the size of attachments, or - in the worst case - to temporarily stop storing attachments if size gets a real issue.
Will the service work with all the mailing list archives?	Yes, because the extraction of the attachment, and the embedding of the link is made before the mails are forwarded to the mailing lists and the archives.
What is the working title of this non-secret project?	We don't know yet ... maybe "Attachmator", or "Libotachment Handler", or - a bit boring - "LibreOffice Mail Attachment Service"?

Mailing List Discussion

You'll find the mail thread discussing this idea on website-discuss (thread starter mail).

Development

Structure and Source-Code

The plan is to implement it using pymime (mailparsing, etc) and django (providing a management-UI for deleting attachments) & hook it up to silverstripe to reuse the existing logins for authentication. The complete rewrite of pymime will provide a multiprocessserver and a flexible plugin architecture. pymime is already used for pre-list-processing (i.e. that is the tool that adds the Archived-At header for mail-archive.com and also adds the footer (and does necessary charset conversions and strips html).

Configuration options and visual theming will allow to adapt a "vanilla" pymime to consider the needs of LibreOffice / The Document Foundation.

The current Dev-Tree is at https://github.com/awerner/pymime/tree/newstyle

The pymime issue tracker is at: https://github.com/awerner/pymime/issues (only LibreOffice / The Document Foundation unrelated issues)

Testing

The service is currently being tested on a separate list solely for testing purposes. When the service runs stable, we will deploy it to more lists. Please note that the current test features a non-themed vanilla version of pymime (means: no LibreOffice / The Document Foundation branding).

Joining the Test:

For testing purposes, you can send an e-mail to
```
test (at) documentfoundation.org
```
where this service is in place now. You need to subscribe to that list first.

Subscribe via

test+subscribe (at) documentfoundation.org

Please send any feedback, questions and errors to
```
postmaster (at) documentfoundation.org 
```
or use the feedback section below.

List archive: https://www.mail-archive.com/test@documentfoundation.org

Configuration

Alex: At the moments, the list "test" is configured like this:

Allowed are mime types
- application/vnd.oasis.opendocument.
- application/x-zip
- application/pdf
- image/gif
- image/png
- image/jpg
- image/svg+xml
and extensions
- .ods
- .odt
- .odp
- .odg
- .pdf
- .gif
- .png
- .jpg
- .jpeg
- .svg
up to a maximum size of the mail of 5M (including the base64-overhead).

Christian:

Different lists can have different settings (different set of allowed attachments, ....)
The time the attachment will be available will be something between 1-6 months.

Known Issues (rarely update list):

The striping of blacklisted attachments leads to the addition of a link to a page showing the mail without attachments. This is a known bug.
An attachment fails if its extension was uppercase (e.g. MAS.PNG) and there is no conversion to lowercase before checking the allowed extensions.
There is no remembering email when time is running out and the attachment will be deleted.

Feedback

Christoph, Question: How to handle localization for the native lang lists? If user on a French list clicks on the link, he might expect to get a localized page (otherwise he might not trust it).

Alex, Answer: The used Webframework, Django, supports l10n via po files. As soon as a libo-theme is created, I'll make the po files available for translation.

Christoph, Wish: If possible, the site should be better themed for "LibreOffice" or "The Document Foundation". For users, this is very important to gain trust that things belong together (at the moment we have the (local) mail client the mail archive and the attachment service looking very differently). The naming "pymime" might be confusing for users (so if it should be kept, I suggest to incorporate it somehow).

Alex, Answer: At the moment, the theme shown is the vanilla-PyMIME theme. It is easy to adapt this theme to the libo/tdf style.

Christoph, Wish: A "front page" would be helpful to explain the rationale of this service (e.g. the Admin link). Maybe even linking to this specification.

Alex, Answer: This is not yet implemented but is on my TODO.

Christoph, Question: Do we need a "disclaimer" stating that the use of this service is a personal risk? At least if we cannot guarantee that the attachments can do some harm.
Christoph, Proposal: If the service runs rather stable, I suggest to have a full usability evaluation - please give me a "ping".
Christoph, Issue: If the attachment expires, then clicking on an attachment link throws an 404 page like this. Instead, we should inform the user that the attachment are not availably anymore and that (s)he should contact the original author of the mail to re-send it.
k-j, Question: Will a stripped off attachment be mentioned in the webpage and will it be noticed there why it is stripped off?

Rollout

Things to not forget when the LibreOffice Mail Attachment Service is enabled officially:

Update TDF Netiquette, Avoid Attachments

Design proposals

by k-j

Comments

The discussion takes place on the website mailing list, see discussion thread start

Information by Rimas Kudelis

Source: Mail by Rimas Kudelies, 2011-03-20

Just FYI: such thing is already implemented for RoundCube's mailing lists. All attachments posted to those lists are stored on http://detached.gigo.com/ instead, and only the links to those attachments are provided in mail footer, like this:

--- 8< --- detachments --- 8< ---
The following attachments have been detached and are available for viewing.
http://detached.gigo.com/rc/RS/pZUugGaU/labels.inc
Only click these links if you trust the sender, as well as this message.
--- 8< --- detachments --- 8< ---

You may want to contact gigo.com's administrator – perhaps he would be willing to share his creation and save LibO from reinventing the wheel.