Development/Budget2017

Proposal for tenders in budget 2017
Below are the ideas for tenders as discussed on the ESC meeting 8th December 2016.

Costs are stated in rough guestimate person weeks, so that it is possible to rank the proposals against each other:

Regular scan-build reports like we do with CppCheck
CostEstimate: 1 week Contact: Luke

Regular scan-build reports like we do with CppCheck. See also: https://wiki.documentfoundation.org/Development/Clang_Code_Analysis

Improve CppCheck
CostEstimate: 2 weeks Contact: Luke


 * Uncheck low priority/quality warnings
 * Show new detected problems since last run

More discussion: http://nabble.documentfoundation.org/Re-Libreoffice-commits-core-git-cppcheck-style-fix-for-noExplicitConstructor-in-writerfilter-tp4201803.html

Query for "last changed by" in Bugzilla
CostEstimate: 8 weeks Contact: Buovjaga

With this much requested feature, we would be able to better track NEEDINFO bugs, self-confirmed bugs (with the ability of "last changed by original reporter" coupled with no one in CC) and who knows what else. It would simply make a lot of things easier for QA.

Antispam feature for Bugzilla (Buovjaga)
CostEstimate: 4 weeks Contact: Buovjaga

During 2016, the QA team had to spend many man-hours fighting spammers. TDF Bugzilla is not alone as all other popular BZs are being hit. A honeypot feature is in development, paid by a certain foundation. I wish TDF would fund a more extensive antispam feature: a framework that allows all sorts of antispam blacklist services to be plugged into BZ. There is some existing work, but not up-to-date or upstream. Bugzilla has agreed to maintain such a feature, if it is contributed.

MSO compatibility test framework using Win API
CostEstimate: 6 weeks + VMs Contact: Tamás Zolnai


 * Microsoft published some API to work with Office applications.
 * https://support.microsoft.com/en-us/kb/196776
 * https://msdn.microsoft.com/en-us/library/microsoft.office.interop.word._document_properties.aspx
 * It can be a good idea to create a test framework which tests MSO file format compatibility by opening files both with LibreOffice and Microsoft Office and checking document properties (e.g. number of pages, shapes, tables) using LO API and Windows API.
 * Of course this test framework would be Windows only.
 * First test can be for example a check for the number of tables in the imported document compared to the table number produced by Microsoft Office import.
 * After this simple test is implemented, we can run this on thousands of documents (this also means we can guarantee a specific level of compatibility by adding these tests and running them on lots of documents (also good for regression testing)).
 * Later the compatibility level can be increased.
 * Another good starting point can be some validation test of LO exported documents.
 * Just open saved documents in MS office automatically and check whether MSO can open them without a problem (e.g. tdf#104787)
 * Validation tools (like Open XML SDK) often shows false positives and also can happen they don't show a problem which actually makes MSO to fail opening a document. So better to test compatibility directly using MSO.

A reasonably future-proof automated testing plan

MSO compatibility test document set
CostEstimate: 3 weeks Contact: Cor Nouws


 * There is an abundance of issues with all kind of interoperability problems.
 * Interoperability is by far the single most reported problem in the real world.
 * The reported issues will have many shared root causes, but often are complex, a mixture and nasty to analyse.
 * Preparing well thought clean test documents will make working on improvements much easier in two ways:
 * Much more appealing to take a look at
 * Saving much developer time; quoting a dev: " It happens quite frequently that creating a minimal reproducer for a bugreport is almost as much time as fixing the actual bug."
 * Focus on OOXML(docx, pptx, xlsx). Binary formats(ppt, doc) to be discussed

A manual document building testing task.

Cleanup & further improve ODF conformance
CostEstimate: 4 weeks Contact: Thorsten Behrens


 * address annoying issues highlighted by http://autotests.opendocumentformat.org/
 * cleanup https://bugs.documentfoundation.org/buglist.cgi?list_id=582656&query_format=advanced&resolution=---&status_whiteboard=odf_validation&status_whiteboard_type=allwordssubstr
 * perhaps also cleanup https://bugs.documentfoundation.org/buglist.cgi?list_id=582655&query_format=advanced&resolution=---&status_whiteboard=odf&status_whiteboard_type=allwordssubstr
 * generally, get LibreOffice ready for ODF 1.3

Deprecate SVG filter in favour of SVGIO
CostEstimate: 2 weeks Contact: Xisco

Discussion -> http://nabble.documentfoundation.org/Two-svg-import-filters-tt4165248.html

ESC Decission -> http://nabble.documentfoundation.org/Libreoffice-qa-minutes-of-ESC-call-tc4171063.html * SVG import filters (Thorsten) + proposal: rip out old Thorsten + Fridrich filter. + Xisco, Regina and Christina kindly helped. + but still not so great at rendering SVGs. + Armin's SVGIO seems better for now => deprecate in favour of svgio; save 2-3k LOC. + still 1-2 gerrit patches there (Michael S)      + bit of a shame. => announce deprecation & removal of it in 5.2 timeframe AI:   + and CC & thank explicitly those who worked on this. (Thorsten)

Image handling re-work
CostEstimate: 8 weeks Contact: Michael

Tackle the well known issues in image handling. Including


 * using a robust, and hard lifecycle mechanism (eg. smart reference count) for every reference to an image - so we never loose an image again
 * propagate this lifecycle mechanism through filters and UNO APIs.
 * copy all (compressed) image streams out of document storages into an on-disk cache - to avoid data loss on file movement.
 * improve image detail reading & storage - to avoid reading a whole JPEG or PNG just to work out its pixel size and discard/swap-out the result
 * reduce excessive swap-in and out thrash
 * ideally - but non-essentially cleanup the "graphics cache size" and manage caching of images in a more intelligent way.

Quite a big swamp here.

split signing from the build process
CostEstimate: 6 weeks Contact: Norbert


 * This should allow signing after the binaries are built rather than during the build process, ideally on a separate signing server.

5.5 idea - re-thinking how we install language-packs
CostEstimate: 8 weeks Contact: Markus


 * if we have an auto-updater with signed MAR files.
 * could provide translated installer, and download rest during install.
 * signing is done on the whole archive with this approach.

unwind EMF+/WMF disaster area
CostEstimate: 12 weeks Contact: Thorsten


 * convert internal use of SVM to scene graph / drawing layer primitives
 * EMF/WMF filter - refactor code to use drawing layer primitives, ala svg import
 * EMF+ filter - remove & convert/re-implement inside EMF/WMF filter code, using drawing layer primitives for rendering the higher-level graphics
 * -> solves a number of sticky problems especially with EMF+, which is especially bad due to it always rendering into bitmaps, even for printing

Unify Writer and Draw Images
CostEstimate: 8 weeks Contact: Thorsten


 * long-standing annoyance
 * several attempts, current status (90-degree rotations) is a half-house
 * extract Writer border rendering & frame special code, move to drawing layer & make accessible from SdrGrafObj

Text layout performance
CostEstimate: 6 weeks Contact: Khaled

Our text layout performance is abysmal, all over the code base it is assumed that shaping text is cheap and we can do it over and over again. Want to measure the text? Shape it and discard the output afterwards. Want to measure part of the same text? Shape again. Want to find line breaks? Shape again. Want to finally draw it? Shape again. This might have been cheap for Latin script in the olden days when all we did is query font cmap table and put glyphs next to each other, which is not the case anymore and never been the case for more involved scripts. We need to work on this, and there are many possibilities; retaining shaping results much longer, improving the wasteful OutputDevice API, caching etc.

Migrating to single modern 2D graphics library
CostEstimate: 20 weeks Contact: Michael

Migrating to single modern 2D graphics library, be it Cairo or Skia, and sorting out our graphics and text drawing inconsistencies.

Curl based HTTP/WebDAV UCP
CostEstimate: 8 weeks Contact: ?

=> implementing a HTTP/WebDAV UCP with Curl, possibly based on code from the Serf UCP, would solve these issues, get rid of 4 bundled external libraries and 1 hard OpenSSL dependency
 * Problem #1: TDF releases bundle both OpenSSL and NSS, both libraries have high number of security issues
 * On macOS and Windows, neither OpenSSL nor NSS integrate with the OS supplied crytographic APIs, so they will use a bundled hard-coded set of trusted CAs that is different from what the OS itself would trust and not user-modifiable
 * OpenSSL cannot ever be used from the system because it has no ABI
 * Problem #2: There are 2 different HTTP/WebDAV UCPs:
 * one (which is used by everybody including TDF releases) using a bundled Neon WebDAV library
 * requires OpenSSL
 * cannot be used on a hypothetical Apple iOS port
 * another (for a hypothetical Apple iOS port) using a bundled Serf library
 * requires OpenSSL
 * Serf does not actually support WebDAV directly, only HTTP, so the UCP itself impleements the additional WebDAV protocol features
 * complicated to build, drags in 2 other bundled external libraries
 * cannot be upgraded to current version without introducing a new build dependency on "scons" tool
 * TDF releases also bundle the Curl library
 * can do HTTP, likely similar to Serf
 * can use native OS cryptographic APIs and trusted CAs on Windows, macOS and Linux
 * can be used on Apple iOS without problems

Native IDE build / simpler building
CostEstimate: 50 weeks Contact: JanI

One of the biggest show-stoppers for keeping new contributors (we have a nice inflow, so no problem there) is our development environment.

Universities today teach students to use IDE, and solely work with that. When I reach out to people who have stopped contributing, the lack of a modern IDE is a major response.

The new “gbuildtojson” that Bjoern has made, is a step in the right direction, but we are still far from a goal, where a contributor clone the repo and open his preferred IDE to start working.

Investing in: - removing cygwin in Windows - making a auto installer (lode++) - complete gbuildtojson and xx-ide-integration to include prepared solutions for popular IDE (msc, Xcode, eclipse)

would help a lot to keep the contributors contributing, and thus help grow the community.

This project has a very wide scope.

More CI hardware to get quicker build-times
CostEstimate: Eur 20k Contact: Noel


 * consider cloud hardware cost; scale on-demand ? (Bjoern)
 * Norbert considers this to be mostly a problem of stopping builds jamming existing hardware

post Macs to people
CostEstimate: Eur 9k Contact: Michael

Lots of core contributions don't have Macs - this makes it really hard to ensure that work which has a cross-platform impact works well on Mac, and (more to the point) that bugs and regressions that affect Macs are addressed. Remote Macs have problems of maintenance, and remote desktop protocols don't work at all well: broken keyboard mappings, and corrupted screens. To solve this we should loan Macs on-site with key developers who make some commitment to look at Mac issues using this as/when they arise. A list of such developers is:


 * Khaled Hosney - low-level text rendering.
 * Michael Meeks - VCL pieces.
 * Jan Holesovsky - various fixing
 * Jan Marek - VCL / main-loop testing
 * Thorsten Behrens - Office Mac to share with Bubli, Armin, Oliver etc.
 * Samuel Mehrbrodt
 * Caolan McNamara - all over the code
 * Michael Stahl - all manner of filters & fixes
 * ... others ? ...

Note 1: Dont we even still have a infra mac idling in the lab?

Note 2: For occasional users, how about shared machine with Chicken Of The VNC Access?

Preconfigured Cloud VMs for Newcomers
CostEstimate: Eur 10k Contact: Bjoern

Have ready-to-go VM with a preinstalled Dev Env including IDE integration with debugging and code completion for the most popular platform (Windows=MSVS, Linux=kdevelop, macOS=XCode) and in the best case a prebuild LibreOffice. Those imagine should be able to be spun up by the dev-mentor at a minutes notice for newcomers to submit their first 3-5 patches. This is asking for budget for the hosting (e.g. cloudonmac or cloudshare) only, I have hope that both volunteers and the dev-mentor will be able to script the setup of these base images.

see also: https://wiki.documentfoundation.org/Hackfests/VMs/Using_a_VM

(This item likely can be folded into the infra budget in the end, which is fine: Then we wouldnt need to vote on this separately.)

finishing online help -> make it actually online
CostEstimate: 6 weeks Contact: Kendy


 * finishing the XHP generating JS, sort out translations,
 * ensure it works off-line with searching; and finally killing help viewer.
 * ideally also online editor (that would upload patches to gerrit)
 * tender submitted for BoD (Olivier)

Collaboration Dictionary Project
CostEstimate: 8 weeks Contact: DennisRoczek

Collaboration Dictionary Project was discussed at I10N: a platform for creating and updating dictionaries and automatically pack extensions from word lists (and upload them to the web pages).

Accessibility improvements
CostEstimate: 2 weeks Contact: Michael

Add tool to find and flag new glade widgets that are added without a11y markup - ie. any new label without a relation for the widget it relates to - should cause a compile / tinderbox warning - except in the case that it is used as a hidden string placeholder to avoid the resource files. Catching the common cases, and black-list all the existing dialogs and/or widgets without these. This should avoid future a11y regressions in the markup that we can scan.

SmartArt implementation
CostEstimate: 16 weeks Contact: Thorsten

Our existing smart-art import uses the fallback-stream in OOX files, and gives us only the draw shapes that are imported - thus loosing the original layout. In older file versions we don't have the cached shapes - so can render nothing. We really would like schema driven diagram layout as a core feature as well. This should be interoperable with OOX, and have suitable extensions for ODF. It should layout interoperably, and allow editing of the underlying data, and selection of a schema.

HSQLDB binary format import
CostEstimate: 6 weeks Contact: Michael

In order to remove the legacy Java / HSQLDB database completely, and move fully to Firebird in 5.4 it is necessary to be able to import old document data with high fidelity from the HSQLDB binary file format which we have used (for performance) inside so many of our existing ODB files. This task involves reading the existing (reasonably simple) Java serialization code, and writing an (import only) filter to import this data safely into base.

Improved built-in scripting debugging
CostEstimate: 10 weeks Contact: Michael

XRay-alike, to be built-in as native code, with better core hooks and usability etc. Modern browsers come with significant, helpful built-in JS developer tools, and we are far behind this in terms of ease-of-use for development of extensions and (more critically) unit tests using the UNO APIs.

Porting the XRay functionality from StarBasic to C++, and including it - along with providing a cleaner, browser-like UI at the bottom of the screen would be ideal.

In particular - the ability to select an item in an arbitrary document eg. an image, or a shape - and get its UNO peer, and be able to inspect attributes on it would be excessively helpful.

Similarly logging UNO events emitted by the core, and allowing inspection of their details in a console-alike.

finish the XServiceDocumenter to allow UNO user to learn about core implementations
CostEstimate: 2 weeks Contact: Bjoern

Finish the com.sun.star.script.XServiceDocumenter/theServiceDocumenter to allow UNO and StarBasic users to use a call to xDocumenter.showCoreDocs to get directly to docs.libreoffice.org showing the C++ classes implementing a specific UNO service. This should smooth the migration of interested people to move from UNO/extension code to core code and scratch their itch.

Note that the showServiceDocs/showInterfaceDocs methods of the service documenter (that bring you to the abstract UNO docs on api.libreoffice.org) are already implemented, so not in scope for this task.

(the XServiceDocumenter.showServiceDocs/showInterfaceDocs could also be a base for "Improved scripting debugging". Possibly merge these?)

see: https://www.youtube.com/watch?v=WBNG6bVZPzw for details

Improve the experience of the SDK
CostEstimate: 4 weeks Contact: Bjoern


 * see https://studiofreya.com/2016/10/11/integrating-libreoffice-into-cpp/ https://studiofreya.com/2016/10/19/errors-connecting-to-libreoffice-with-cpp-errors-part-2/ https://studiofreya.com/2016/10/21/open-libreoffice-calc-with-cpp-part-3/ as reference of the pain this currently is
 * undo hugely painful gnumake-ness etc.
 * make it much more usable, and ideally from IDEs
 * provide a 10 minutes "Hello World extension for LibreOffice from scratch" video as proof how simple stuff is after this is fixed.

UNO API changes subscription/feedback channel
CostEstimate: 1 week Contact: Bjoern

see: http://nabble.documentfoundation.org/minutes-of-ESC-call-tp4200924p4202871.html

The simplest solution (KISS for a start) would be e.g.:
 * an archived mailing list (uno-changes@ ...)
 * an bot that puts every API change on that list, cc'ing the author of the change,
 * someone replying to the API change also reaches the author of the change
 * bot that filters for the API change discussions and put notifications and RSS feeds on api.libreoffice.org

UNO API discussion/sniplet dump
CostEstimate: 4 weeks Contact: Bjoern

PHP is a horrible language, still it succeeded in capturing a huge amount of the market. I assume, the "user contributed notes" on their API documentation played a major part in that (see e.g.: http://php.net/manual/en/function.strlen.php). This proposal is to add a similar user-contributed code examples directly to a specific part of the UNO API documentation at api.libreoffice.org. The discussion software should have a low barrier to get into (OpenID?) and support e.g. code highlighting. Askbot might be a candidate.

This is pretty much a superset of the "UNO API changes subscription/feedback channel" propoal: those bots and RSS feeds could use the discussion forum.

User Metrics
CostEstimate: 12 weeks Contact: Kendy

Re-run the User Metrics tender as before.

[ https://blog.documentfoundation.org/blog/2015/02/24/tender-to-develop-and-incorporate-usability-metrics-collection-for-libreoffice-201502-02/ ]

Redesign of chart wizard UI
CostEstimate: 6 weeks Contact: Bubli


 * Redesign of chart wizard UI, pretty sexy mockups from Heiko Tietze here: http://user-prompt.com/libreoffice-design-session-inserting-a-chart/
 * Related to that - chart styles. Was a part of this OSBA thingit, but never found funding: http://osb-alliance.de/fileadmin/Working_Groups/OfficeInteroperability/Project2/SpecificationMajorFeatureImprovements_final.pdf

Improving printing UI
CostEstimate: 20 weeks Contact: Bubli


 * Improving printing UI, more sexy mockup from Heiko here: http://pad.documentfoundation.org/p/UX-PrintDialog

Automatic Detection and Bisection of Regressions
'''CostEstimate: in-progess Contact: Luke Document and improve the https://github.com/milossramek/office-interoperability-tools tools described at https://www.youtube.com/watch?v=cfzMP4oTZwo for the QA team. Add into LO QA framework. Have an end goal of reaching a state where it could run automatic weekly reports against a pool of test documents. Xisco: I'm already working on it: https://github.com/x1sc0/office-interoperability-tools

ongoing

could we have a 'tips' scheme
CostEstimate: ? Contact: Heiko


 * KDE side, use pay-pal only https://www.kde.org/fundraisers/yearend2016/
 * sounds like re-inventing freedom sponsors (Bubli)
 * if this happens - do it outside the foundation to avoid issues (Norbert)
 * like barnstars but with a financial 'tip' - is the idea.

abandoned - Norbert (TDF treasurer) is not a fan; so lets do it outside TDF funding.

patch update code
CostEstimate: unclear yet / just for Mac? Contact: Markus

=> not sure it will work out as a tender; already 70% done.
 * allow pushing patches, lots of details to sort out
 * have a FOSDEM talk for this in the dev-room
 * talk to releng & devs there @ the hack-fest.
 * Windows & Linux ~done; no Mac so not tested

defer - until after FOSDEM.

Subpixel and stable glyph positioning
Contact: Khaled Fixing the bad text spacing we have due to lack of subpixel positioning. All graphics system we use (except GDI, but this is dying) support subpixel positioning but we make no use of it as all our glyphs positions are stored using integers (not to mention the rounding errors all over the place). This results in very bad text spacing and it gets worse on hidpi screens. Thinking about this more, I think it can be a GSoC project.

Add Impress remote capabilities for simple ad-hoc fingerpainting ad-hoc sketches
Contact: Bjoern Likely better suited to GSOC -- adding it here as we collect budget extensions. The idea is basically to allow to do simple ad-hoc sketches on a tablet with the impress remote, e.g. to answer an audience question on the spot. Allowing the tablet to use the projector as a modern chalkboard.

Make Calc/Base/Chart handle small set of geographic data
Allow Calc/Base/Chart to handle regional subsets of geographic data, e.g. import/edit/draw a small region of OpenStreetMap data.

see also: https://blog.documentfoundation.org/blog/2016/09/19/prototypefund-an-opportunity-for-german-freelance-developers/