Tenders for budget 2018

    From The Document Foundation Wiki

    TDF LibreOffice Document Liberation Project Community Blogs Weblate Nextcloud Redmine Ask LibreOffice Donate


    Proposal for tenders in budget 2018

    Below are proposal for tenders, to be discussed on the ESC.

    Costs are stated in rough guestimate person weeks, so that it is possible to rank the proposals against each other:

    QA and Automated Testing

    Regular scan-build reports like we do with CppCheck

    CostEstimate: 1 week Contact: Luke

    Regular scan-build reports like we do with CppCheck. See also: https://wiki.documentfoundation.org/Development/Clang_Code_Analysis

    Improve CppCheck

    CostEstimate: 2 weeks Contact: Luke

    • Uncheck low priority/quality warnings
    • Fix false positives caused by incorrect configuration
    • Show new detected problems since last run

    MSO compatibility test framework using Win API

    CostEstimate: 6 weeks + VMs Contact: Tamás Zolnai

    • Microsoft published some API to work with Office applications.
    • It can be a good idea to create a test framework which tests MSO file format compatibility by opening files both with LibreOffice and Microsoft Office and checking document properties (e.g. number of pages, shapes, tables) using LO API and Windows API.
      • Of course this test framework would be Windows only.
    • First test can be for example a check for the number of tables in the imported document compared to the table number produced by Microsoft Office import.
      • After this simple test is implemented, we can run this on thousands of documents (this also means we can guarantee a specific level of compatibility by adding these tests and running them on lots of documents (also good for regression testing)).
      • Later the compatibility level can be increased.
    • Another good starting point can be some validation test of LO exported documents.
      • Just open saved documents in MS office automatically and check whether MSO can open them without a problem (e.g. tdf#104787)
      • Validation tools (like Open XML SDK) often shows false positives and also can happen they don't show a problem which actually makes MSO to fail opening a document. So better to test compatibility directly using MSO.

    A reasonably future-proof automated testing plan

    MSO compatibility test document set

    CostEstimate: 3 weeks Contact: Cor Nouws

    • There is an abundance of issues with all kind of interoperability problems.
    • Interoperability is by far the single most reported problem in the real world.
    • The reported issues will have many shared root causes, but often are complex, a mixture and nasty to analyse.
    • Preparing well thought clean test documents will make working on improvements much easier in two ways:
      • Much more appealing to take a look at
      • Saving much developer time; quoting a dev: " It happens quite frequently that creating a minimal reproducer for a bugreport is almost as much time as fixing the actual bug."
    • Focus (docx, pptx ... xlsx ... ppt, doc) to be discussed

    A manual document building testing task.

    Cleanup & further improve ODF conformance

    CostEstimate: 4 weeks Contact: Thorsten Behrens

    Automated ODF filter regression testing

    CostEstimate: 3 weeks Contact: Michael Stahl

    There are sometimes dataloss regression bugs with the filters for LO's default file format, ODF; the unit tests have nowhere near sufficient coverage to prevent this. The proposal is to install an early-warning system for such problems. The idea is to use the ODFunDiff tool, which was developed specifically with relevant (functional and performance) requirements in mind; for details see https://conference.libreoffice.org/assets/Conference/Brno/stahl-functional-document-testing.pdf

    • It may be necessary to do a few tweaks and updates to the tool, as it was originally developed in the LO 5.2/5.3 time.
    • It may be necessary to fix some hypothetical additional non-determinism in LO that may have crept in since LO 5.3.
    • Some scripting needs to be developed to extend the weekly crash-report run with an additional report about detected ODF differences on the ~27k ODF round-trips
    • The test should use last run's ODF files as reference to compare against
    • Likely, resource problems will need to be dealt with: crashtesting VM doesn't have enough disk space for 2 full sets of result documents

    Core Code Refactorings and Cleanups

    convert Impress slideshow to drawinglayer primitives

    CostEstimate: 4 weeks Contact: Thorsten

    This is a copy from Development/GSoC/Ideas#Rework_Impress_slideshow_to_use_DrawingLayer_primitives, which has not happend in 4 years.

    The Impress slideshow, while being designed to only interact with Impress via interfaces, had to resort to an ugly hack to be able to render all Impress content. That was ok back in the day, but is becoming a liability these days. Nowadays, what one want to use is the DrawingLayer Primitives (https://wiki.openoffice.org/wiki/DrawingPrimitives), which means porting over slideshow/source/engine/shapes/* to use this kind of abstraction, instead of the StarView Metafile previously in use.

    split signing from the build process

    CostEstimate: 6 weeks Contact: Norbert

    • This should allow signing after the binaries are built rather than during the build process, ideally on a separate signing server.

    Re-thinking how we install language-packs

    CostEstimate: 8 weeks Contact: Markus

    • if we have an auto-updater with signed MAR files.
    • could provide translated installer, and download rest during install.
    • signing is done on the whole archive with this approach.

    Unify Writer and Draw Images

    CostEstimate: 8 weeks Contact: Thorsten

    • long-standing annoyance
    • several attempts, current status (90-degree rotations) is a half-house. Rotation issue properly fixed by Armin in 6.0.
    • extract Writer border rendering & frame special code, move to drawing layer & make accessible from SdrGrafObj

    Implement font subsetter for font embedding

    CostEstimate: 4 weeks Contact: Thorsten

    While font embedding is implemented for LibreOffice since many years, it's an under-used and under-developed feature due to the fact that unicode-capable fonts are massive, thus blowing up documents beyond useful sizes. Other applications therefore simply subset the fonts embedded, to either exactly the set of glyphs used, or to the code pages / script areas touched by existing document content. This would also help a lot for making our in-tree and out-of-tree bug document collection invariant to installed system fonts.

    Text layout performance

    CostEstimate: 6 weeks Contact: Khaled

    Our text layout performance is abysmal, all over the code base it is assumed that shaping text is cheap and we can do it over and over again. Want to measure the text? Shape it and discard the output afterwards. Want to measure part of the same text? Shape again. Want to find line breaks? Shape again. Want to finally draw it? Shape again. This might have been cheap for Latin script in the olden days when all we did is query font cmap table and put glyphs next to each other, which is not the case anymore and never been the case for more involved scripts. We need to work on this, and there are many possibilities; retaining shaping results much longer, improving the wasteful OutputDevice API, caching etc.

    Migrating to single modern 2D graphics library

    CostEstimate: 20 weeks Contact: Michael

    Migrating to single modern 2D graphics library, be it Cairo or Skia, and sorting out our graphics and text drawing inconsistencies.

    Curl based HTTP/WebDAV UCP

    CostEstimate: 8 weeks Contact: Michael S

    • Problem #1: TDF releases bundle both OpenSSL and NSS, both libraries have high number of security issues
      • On macOS and Windows, neither OpenSSL nor NSS integrate with the OS supplied crytographic APIs, so they will use a bundled hard-coded set of trusted CAs that is different from what the OS itself would trust and not user-modifiable
      • OpenSSL cannot ever be used from the system because it has no ABI
    • Problem #2: There are 2 different HTTP/WebDAV UCPs:
      • one (which is used by everybody including TDF releases) using a bundled Neon WebDAV library
        • requires OpenSSL
        • cannot be used on a hypothetical Apple iOS port
      • another (for a hypothetical Apple iOS port) using a bundled Serf library
        • requires OpenSSL
        • Serf does not actually support WebDAV directly, only HTTP, so the UCP itself impleements the additional WebDAV protocol features
        • complicated to build, drags in 2 other bundled external libraries
        • cannot be upgraded to current version without introducing a new build dependency on "scons" tool
    • TDF releases also bundle the Curl library
      • can do HTTP, likely similar to Serf
      • can use native OS cryptographic APIs and trusted CAs on Windows, macOS and Linux
      • can be used on Apple iOS without problems

    => implementing a HTTP/WebDAV UCP with Curl, possibly based on code from the Serf UCP, would solve these issues, get rid of 4 bundled external libraries and 1 hard OpenSSL dependency

    AW080 Paradigm changes: SdrObject Homogen Transformations

    CostEstimate: 8 weeks Contact: Armin

    For general information of ˜AW080 paradigm changes see [1]. This single Paradigm change is about using Homogen Transformations (Linear Algebra) throughout the DrawingLayer Model definitions (mainly but not only SdrObjects). I was doing this once for AW080 and know how this can work and what is doable. For a discussion about advantages of doing this, see the text following the link.

    AW080: Removal of NBC Methods, Improve DrawingLayer Broadcasting

    CostEstimate: 2 weeks Contact: Armin

    For general information of ˜AW080 paradigm changes see [2]. This Paradigm change is about getting rid of the old NBC-Methods (which stands for ˜NonBroadCast BTW) in DrawingLayer and cleanup of the Broadcasting/Messaging part of DrawingLayer involved with that. I was doing this once for AW080 and know how this can work and what is doable. For a discussion about advantages of doing this, see the text following the link.

    AW080 Precision changes: Enhance DrawingLayer Precision

    CostEstimate: 4 weeks Contact: Armin

    For general information of ˜AW080 precision changes see [3]. There are quite some Precision Changes which should be done, not all of these will be doable in the mentioned time frame. This is a more general entry to suggest spending some time on this and do as much as possible of the points mentioned in the Wiki page. For a discussion off possibilities see the text following the link.

    AW080 Other Changes: Various DrawingLayer enhancements

    CostEstimate: 4 weeks Contact: Armin

    For general information of ˜AW080 Other changes see [4]. There are many Other Changes/enhancements possible (called so because not fitting to the other categories), not all of these will be doable in the mentioned time frame. This is a more general point to suggest spending some time on this and do as much as possible of the mentioned points. For a discussion off possibilities see the text following the link.

    Development Infrastructure

    Native IDE build / simpler building

    CostEstimate: 50 weeks Contact: JanI

    One of the biggest show-stoppers for keeping new contributors (we have a nice inflow, so no problem there) is our development environment.

    Universities today teach students to use IDE, and solely work with that. When I reach out to people who have stopped contributing, the lack of a modern IDE is a major response.

    The new “gbuildtojson” that Bjoern has made, is a step in the right direction, but we are still far from a goal, where a contributor clone the repo and open his preferred IDE to start working.

    Investing in: - removing cygwin in Windows - making a auto installer (lode++) - complete gbuildtojson and xx-ide-integration to include prepared solutions for popular IDE (msc, Xcode, eclipse)

    would help a lot to keep the contributors contributing, and thus help grow the community.

    This project has a very wide scope.

    More CI hardware to get quicker build-times

    CostEstimate: Eur 20k Contact: Noel

    • consider cloud hardware cost; scale on-demand ? (Bjoern)
    • Norbert considers this to be mostly a problem of stopping builds jamming existing hardware

    Preconfigured Cloud VMs for Newcomers

    CostEstimate: Eur 10k Contact: Bjoern

    Have ready-to-go VM with a preinstalled Dev Env including IDE integration with debugging and code completion for the most popular platform (Windows=MSVS, Linux=kdevelop, macOS=XCode) and in the best case a prebuild LibreOffice. Those imagine should be able to be spun up by the dev-mentor at a minutes notice for newcomers to submit their first 3-5 patches. This is asking for budget for the hosting (e.g. cloudonmac or cloudshare) only, I have hope that both volunteers and the dev-mentor will be able to script the setup of these base images.

    see also: https://wiki.documentfoundation.org/Hackfests/VMs/Using_a_VM

    (This item likely can be folded into the infra budget in the end, which is fine: Then we wouldnt need to vote on this separately.)

    Collaboration Dictionary Project

    CostEstimate: 8 weeks Contact: DennisRoczek

    Collaboration Dictionary Project was discussed at I10N: a platform for creating and updating dictionaries and automatically pack extensions from word lists (and upload them to the web pages).

    CI for Online

    CostEstimate: ? Contact: Samuel M.

    There should be a Jenkins job that runs "make check" for each gerrit patch for LO Online, as we do for LO core. https://redmine.documentfoundation.org/issues/2541

    Actually, the same is true for Android...

    Blocking features

    SmartArt implementation

    CostEstimate: 16 weeks Contact: Thorsten

    Our existing smart-art import uses the fallback-stream in OOX files, and gives us only the draw shapes that are imported - thus loosing the original layout. In older file versions we don't have the cached shapes - so can render nothing. We really would like schema driven diagram layout as a core feature as well. This should be interoperable with OOX, and have suitable extensions for ODF. It should layout interoperably, and allow editing of the underlying data, and selection of a schema.

      • Note** this has been significantly improved already during GSoC 2017, possibly obsolete, at the very least much reduced effort

    Improved built-in, XRay-like profiler & script debug

    CostEstimate: 12 weeks Contact: Michael

    XRay-alike, to be built-in as native code, with better core hooks and usability etc. Modern browsers come with significant, helpful built-in JS developer tools, and we are far behind this in terms of ease-of-use for development of extensions and (more critically) unit tests using the UNO APIs.

    It should be possible to press F12 (or whatever) - and get access to the rich object model, browse it, and see how both scripts, and the app is performing - to help both scripters, and also those interested in improving LibreOffice performance & responsiveness.

    Porting the XRay functionality from StarBasic to C++, and including it - along with providing a cleaner, browser-like UI at the bottom of the screen would be ideal.

    In particular - the ability to select an item in an arbitrary document eg. an image, or a shape - and get its UNO peer, and be able to inspect attributes on it would be excessively helpful.

    Similarly logging UNO events emitted by the core, and allowing inspection of their details in a console-alike.

    finish XServiceDocumenter so UNO user can learn about core implementations

    CostEstimate: 2 weeks Contact: Bjoern

    Finish the com.sun.star.script.XServiceDocumenter/theServiceDocumenter to allow UNO and StarBasic users to use a call to xDocumenter.showCoreDocs() to get directly to docs.libreoffice.org showing the C++ classes implementing a specific UNO service. This should smooth the migration of interested people to move from UNO/extension code to core code and scratch their itch.

    Note that the showServiceDocs()/showInterfaceDocs() methods of the service documenter (that bring you to the abstract UNO docs on api.libreoffice.org) are already implemented, so not in scope for this task.

    (the XServiceDocumenter.showServiceDocs()/showInterfaceDocs() could also be a base for "Improved scripting debugging". Possibly merge these?)

    see: https://www.youtube.com/watch?v=WBNG6bVZPzw for details

    Improve the experience of the SDK

    CostEstimate: 4 weeks Contact: Bjoern

    UNO API changes subscription/feedback channel

    CostEstimate: 1 week Contact: Bjoern

    The simplest solution (KISS for a start) would be e.g.:

    • an archived mailing list (uno-changes@ ...)
    • an bot that puts every API change on that list, cc'ing the author of the change,
      • someone replying to the API change also reaches the author of the change
    • bot that filters for the API change discussions and put notifications and RSS feeds on api.libreoffice.org

    UNO API discussion/sniplet dump

    CostEstimate: 4 weeks Contact: Bjoern

    PHP is a horrible language, still it succeeded in capturing a huge amount of the market. I assume, the "user contributed notes" on their API documentation played a major part in that (see e.g.: http://php.net/manual/en/function.strlen.php). This proposal is to add a similar user-contributed code examples directly to a specific part of the UNO API documentation at api.libreoffice.org. The discussion software should have a low barrier to get into (OpenID?) and support e.g. code highlighting. Askbot might be a candidate.

    This is pretty much a superset of the "UNO API changes subscription/feedback channel" propoal: those bots and RSS feeds could use the discussion forum.

    Add Winding-Rule support for PolyPolygon Handling

    CostEstimate: 2 weeks Contact: Armin

    This task suggests to use the Winding-Rule for PolyPolygons inside and throughout the Office. The Winding-Rule defines how a PolyPolygon is to be graphically represented (see [5]. We currently have no parameter associated with our PolyPolygons to represent that and imply these to follow the Nonzero-Rule This gets problematic with various external Formats supporting both Winding-Rules. In the best cases we try to 'Convert' imported PolyPolygons which are not of type Nonzero-Rule to this. This is very runtime-expensive (see some very slow loading imports) and requires fully cut and touch free PolyPolygons, including the whole clipping steps. For export, we always assume Nonzero-Rule because we know no other. This change is about supporting both Winding-Rules at Imports, in the Core, in the graphic stacks (most support it anyways) and at export time (also supported by quite some formats). It is not needed or planned to offer at the UI (from my POV).

    Support Multi-Step-Gradients and 'SVG-Gradients' in LO

    CostEstimate: 10 weeks Contact: Armin

    One highly missing feature are Multi-Step-Gradients. This shows in quite some places, e.g. at import of external formats which support these. We have to reduce to Two-Color-Gradients which are as 'similar' as possible. This is also true for the gradient geometry we have to use one of our old internal formats that comes as close as possible to more modern formats. This is also true for SVG-Gradients currently. We should be able to support these directly and add these to our internal gradients. These changes have quite some aspects of which not all will be solvable in one step. It should include: - Support in core (model/rendering) - Support in FileFormats (ODF for MultiColorSteps, SVG-Gradients, partial adaption of external format im/export filters) - Support in UI (MultiColorSteps for existing and SVG-Gradients – not sure how lean this can be, probably no initial support for interactive Gradients – yes we have something like this already)

    Direct Conversion from Primitives to SdrObjects

    CostEstimate: 2 weeks Contact: Armin

    We currently have no conversion step to create our own internal Graphic formats using SdrObjects when the source are Primitives. This is e.g. the case with SVG imports and other vector formats. These are currently converted to SdrObjects using an indirection step: First convert Primitives to Metafile (with all known internal limitations and errors of that old format) and then using pretty old code to convert that to SdrObjects. This indirection loses quality (integer) and performance. Solution would be to have one single internal direct converter that is capable of creating SdrObjects directly from Primitives which will be used in all importers of vector formats – existing and future ones). This would e.g. be the precondition to have/support SVG-Gradients one day directly in LO (see below).

    Direct Creation of CustomShapeGeometry using Primitives

    CostEstimate: 4 weeks Contact: Armin

    Currently the CustomShapes (also known as AutoShapes) internally create their visualization using 'hidden' SdrObjects from the DrawingLayer, being e.g. non 'real'members of the SdrModel. This always leads to problems in many areas, including many 'hacks' avoiding nullptr accesses and more. The usage of these 'hidden' SdrObjects is to create the needed Primitives from these. This indirection step costs runtime and ressources, loses quality (by being on 100thmm integer) and is partially problematic - this is especially true for complex CustomShapes that have many Faces (which still is not error-free nowadays due to limits in what SdrObjects can graphically represent). This would also fix runtime/quality problems with extruded CustomShapes which use the MS-like 3D visualizations. These do very special 'tricks' to represent all this using 3D-SdrObjects. Using and creating Primitive representations directly would make things much faster and reliable.

    UX problem areas

    User Metrics

    CostEstimate: 12 weeks Contact: Kendy

    Re-run the User Metrics tender as before.

    https://blog.documentfoundation.org/blog/2015/02/24/tender-to-develop-and-incorporate-usability-metrics-collection-for-libreoffice-201502-02/

    Suggestion from Samuel: Limit the client side implementation as the scope is quite big already. New client metrics can be added by the community once the infrastructure is there. Modify the list like this e.g.:

    3. a client part, that not only counts how often features have been used, but also provides further metrics; some samples of items that need tracking are

    • the location of the click action in the menu, as sometimes duplicates exist
    • which context menu was used
    • whether a certain feature was invoked by a single click, by click and hold, by a drop down click or by a multi click
    • from which application clicking on the close document/window ‘X’ (.CloseDoc) and close application ‘X’ (.Quit) occurs
    • whether the user used the enter key, mouse click or an accelerator to open a menu item
    • how the app was opened (via command line, start menu, start center, or by opening a document)
    • in which toolbar a button was clicked, as some buttons are in multiple toolbars, and users can add buttons to toolbars individually
    • which slide transitions and object animations are used most in Impress
    • the concrete action/command sequence: which action was used by the user, and which was the next action used after that (e.g. inserting an image and then adding a caption)
    • which menu bar keystroke sequences are used (e.g. Alt+F + O)
    • which icon theme, font list and theme name the user has active

    Redesign of chart wizard UI

    CostEstimate: 6 weeks Contact: Bubli

    Improving printing UI

    CostEstimate: 20 weeks Contact: Bubli

    Already Resourced / Abandoned

    Automatic Detection and Bisection of Regressions

    CostEstimate: in-progess Contact: Luke Document and improve the https://github.com/milossramek/office-interoperability-tools tools described at https://www.youtube.com/watch?v=cfzMP4oTZwo for the QA team. Add into LO QA framework. Have an end goal of reaching a state where it could run automatic weekly reports against a pool of test documents. Xisco: I'm already working on it: https://github.com/x1sc0/office-interoperability-tools

    UPDATE 02-22-2018 (Xisco) The office-interoperability tool is already working in a vm testing rtf, doc, docx, odt (export), ppt, pptx and odp (export) against master builds using 8,450 files in total. The issues found by the tool can be found here: https://bugs.documentfoundation.org/buglist.cgi?list_id=778569&longdesc=office-interoperability-tools&longdesc_type=casesubstring&query_format=advanced

    could we have a 'tips' scheme

    CostEstimate: ? Contact: Heiko

    • KDE side, use pay-pal only https://www.kde.org/fundraisers/yearend2016/
    • sounds like re-inventing freedom sponsors (Bubli)
    • if this happens - do it outside the foundation to avoid issues (Norbert)
    • like barnstars but with a financial 'tip' - is the idea.

    abandoned - Norbert (TDF treasurer) is not a fan; so lets do it outside TDF funding.

    patch update code

    CostEstimate: unclear yet / just for Mac? Contact: Markus

    • allow pushing patches, lots of details to sort out
    • have a FOSDEM talk for this in the dev-room
    • talk to releng & devs there @ the hack-fest.

    => not sure it will work out as a tender; already 70% done.

    • Windows & Linux ~done; no Mac so not tested

    defer - until after FOSDEM.

    Proposals missing detailed descriptions for tenders in budget 2018

    Better integration of extensions (Heiko)

    Design-team page of topics for GSOC (Heiko)

    Proposals for tenders in budget 2018 that are likely better suited as a GSOC

    Subpixel and stable glyph positioning

    Contact: Khaled Fixing the bad text spacing we have due to lack of subpixel positioning. All graphics system we use (except GDI, but this is dying) support subpixel positioning but we make no use of it as all our glyphs positions are stored using integers (not to mention the rounding errors all over the place). This results in very bad text spacing and it gets worse on hidpi screens. Thinking about this more, I think it can be a GSoC project.

    Add Impress remote capabilities for simple ad-hoc fingerpainting ad-hoc sketches

    Contact: Bjoern Likely better suited to GSOC -- adding it here as we collect budget extensions. The idea is basically to allow to do simple ad-hoc sketches on a tablet with the impress remote, e.g. to answer an audience question on the spot. Allowing the tablet to use the projector as a modern chalkboard.

    Make Calc/Base/Chart handle small set of geographic data

    Allow Calc/Base/Chart to handle regional subsets of geographic data, e.g. import/edit/draw a small region of OpenStreetMap data.

    see also: https://blog.documentfoundation.org/blog/2016/09/19/prototypefund-an-opportunity-for-german-freelance-developers/

    Retrieved from "https://wiki.documentfoundation.org/index.php?title=Development/Budget2018&oldid=643651"