Tenders for budget 2017

    From The Document Foundation Wiki

    TDF LibreOffice Document Liberation Project Community Blogs Weblate Nextcloud Redmine Ask LibreOffice Donate


    Proposal for tenders in budget 2017

    Below are the ideas for tenders as discussed on the ESC meeting 8th December 2016.

    Costs are stated in rough guestimate person weeks, so that it is possible to rank the proposals against each other:

    QA and Automated Testing

    Regular scan-build reports like we do with CppCheck

    CostEstimate: 1 week Contact: Luke

    Regular scan-build reports like we do with CppCheck. See also: https://wiki.documentfoundation.org/Development/Clang_Code_Analysis

    Improve CppCheck

    CostEstimate: 2 weeks Contact: Luke

    • Uncheck low priority/quality warnings
    • Show new detected problems since last run


    Query for "last changed by" in Bugzilla

    CostEstimate: 8 weeks Contact: Buovjaga

    With this much requested feature, we would be able to better track NEEDINFO bugs, self-confirmed bugs (with the ability of "last changed by original reporter" coupled with no one in CC) and who knows what else. It would simply make a lot of things easier for QA.

    Antispam feature for Bugzilla (Buovjaga)

    CostEstimate: 4 weeks Contact: Buovjaga

    During 2016, the QA team had to spend many man-hours fighting spammers. TDF Bugzilla is not alone as all other popular BZs are being hit. A honeypot feature is in development, paid by a certain foundation. I wish TDF would fund a more extensive antispam feature: a framework that allows all sorts of antispam blacklist services to be plugged into BZ. There is some existing work, but not up-to-date or upstream. Bugzilla has agreed to maintain such a feature, if it is contributed.

    MSO compatibility test framework using Win API

    CostEstimate: 6 weeks + VMs Contact: Tamás Zolnai

    • Microsoft published some API to work with Office applications.
    • It can be a good idea to create a test framework which tests MSO file format compatibility by opening files both with LibreOffice and Microsoft Office and checking document properties (e.g. number of pages, shapes, tables) using LO API and Windows API.
      • Of course this test framework would be Windows only.
    • First test can be for example a check for the number of tables in the imported document compared to the table number produced by Microsoft Office import.
      • After this simple test is implemented, we can run this on thousands of documents (this also means we can guarantee a specific level of compatibility by adding these tests and running them on lots of documents (also good for regression testing)).
      • Later the compatibility level can be increased.
    • Another good starting point can be some validation test of LO exported documents.
      • Just open saved documents in MS office automatically and check whether MSO can open them without a problem (e.g. tdf#104787)
      • Validation tools (like Open XML SDK) often shows false positives and also can happen they don't show a problem which actually makes MSO to fail opening a document. So better to test compatibility directly using MSO.

    A reasonably future-proof automated testing plan

    MSO compatibility test document set

    CostEstimate: 3 weeks Contact: Cor Nouws

    • There is an abundance of issues with all kind of interoperability problems.
    • Interoperability is by far the single most reported problem in the real world.
    • The reported issues will have many shared root causes, but often are complex, a mixture and nasty to analyse.
    • Preparing well thought clean test documents will make working on improvements much easier in two ways:
      • Much more appealing to take a look at
      • Saving much developer time; quoting a dev: " It happens quite frequently that creating a minimal reproducer for a bugreport is almost as much time as fixing the actual bug."
    • Focus on OOXML(docx, pptx, xlsx). Binary formats(ppt, doc) to be discussed

    A manual document building testing task.

    Cleanup & further improve ODF conformance

    CostEstimate: 4 weeks Contact: Thorsten Behrens

    Core Code Refactorings and Cleanups

    Deprecate SVG filter in favour of SVGIO

    CostEstimate: 2 weeks Contact: Xisco

    ESC Decission 

    * SVG import filters (Thorsten)
   + proposal: rip out old Thorsten + Fridrich filter.
      + Xisco, Regina and Christina kindly helped.
         + but still not so great at rendering SVGs.
   + Armin's SVGIO seems better for now
      => deprecate in favour of svgio; save 2-3k LOC.
   + still 1-2 gerrit patches there (Michael S)
      + bit of a shame.
   => announce deprecation & removal of it in 5.2 timeframe
AI:    + and CC & thank explicitly those who worked on this. (Thorsten)

    Image handling re-work

    CostEstimate: 8 weeks Contact: Michael

    Tackle the well known issues in image handling [1]. Including

    • using a robust, and hard lifecycle mechanism (eg. smart reference count) for every reference to an image - so we never loose an image again
    • propagate this lifecycle mechanism through filters and UNO APIs.
    • copy all (compressed) image streams out of document storages into an on-disk cache - to avoid data loss on file movement.
    • improve image detail reading & storage - to avoid reading a whole JPEG or PNG just to work out its pixel size and discard/swap-out the result
    • reduce excessive swap-in and out thrash
    • ideally - but non-essentially cleanup the "graphics cache size" and manage caching of images in a more intelligent way.

    Quite a big swamp here.

    split signing from the build process

    CostEstimate: 6 weeks Contact: Norbert

    • This should allow signing after the binaries are built rather than during the build process, ideally on a separate signing server.

    5.5 idea - re-thinking how we install language-packs

    CostEstimate: 8 weeks Contact: Markus

    • if we have an auto-updater with signed MAR files.
    • could provide translated installer, and download rest during install.
    • signing is done on the whole archive with this approach.

    unwind EMF+/WMF disaster area

    CostEstimate: 12 weeks Contact: Thorsten

    • convert internal use of SVM to scene graph / drawing layer primitives
    • EMF/WMF filter - refactor code to use drawing layer primitives, ala svg import
    • EMF+ filter - remove & convert/re-implement inside EMF/WMF filter code, using drawing layer primitives for rendering the higher-level graphics
    • -> solves a number of sticky problems especially with EMF+, which is especially bad due to it always rendering into bitmaps, even for printing

    Unify Writer and Draw Images

    CostEstimate: 8 weeks Contact: Thorsten

    • long-standing annoyance
    • several attempts, current status (90-degree rotations) is a half-house
    • extract Writer border rendering & frame special code, move to drawing layer & make accessible from SdrGrafObj

    Text layout performance

    CostEstimate: 6 weeks Contact: Khaled

    Our text layout performance is abysmal, all over the code base it is assumed that shaping text is cheap and we can do it over and over again. Want to measure the text? Shape it and discard the output afterwards. Want to measure part of the same text? Shape again. Want to find line breaks? Shape again. Want to finally draw it? Shape again. This might have been cheap for Latin script in the olden days when all we did is query font cmap table and put glyphs next to each other, which is not the case anymore and never been the case for more involved scripts. We need to work on this, and there are many possibilities; retaining shaping results much longer, improving the wasteful OutputDevice API, caching etc.

    Migrating to single modern 2D graphics library

    CostEstimate: 20 weeks Contact: Michael

    Migrating to single modern 2D graphics library, be it Cairo or Skia, and sorting out our graphics and text drawing inconsistencies.

    Curl based HTTP/WebDAV UCP

    CostEstimate: 8 weeks Contact: ?

    • Problem #1: TDF releases bundle both OpenSSL and NSS, both libraries have high number of security issues
      • On macOS and Windows, neither OpenSSL nor NSS integrate with the OS supplied crytographic APIs, so they will use a bundled hard-coded set of trusted CAs that is different from what the OS itself would trust and not user-modifiable
      • OpenSSL cannot ever be used from the system because it has no ABI
    • Problem #2: There are 2 different HTTP/WebDAV UCPs:
      • one (which is used by everybody including TDF releases) using a bundled Neon WebDAV library
        • requires OpenSSL
        • cannot be used on a hypothetical Apple iOS port
      • another (for a hypothetical Apple iOS port) using a bundled Serf library
        • requires OpenSSL
        • Serf does not actually support WebDAV directly, only HTTP, so the UCP itself impleements the additional WebDAV protocol features
        • complicated to build, drags in 2 other bundled external libraries
        • cannot be upgraded to current version without introducing a new build dependency on "scons" tool
    • TDF releases also bundle the Curl library
      • can do HTTP, likely similar to Serf
      • can use native OS cryptographic APIs and trusted CAs on Windows, macOS and Linux
      • can be used on Apple iOS without problems

    => implementing a HTTP/WebDAV UCP with Curl, possibly based on code from the Serf UCP, would solve these issues, get rid of 4 bundled external libraries and 1 hard OpenSSL dependency

    Development Infrastructure

    Native IDE build / simpler building

    CostEstimate: 50 weeks Contact: JanI

    One of the biggest show-stoppers for keeping new contributors (we have a nice inflow, so no problem there) is our development environment.

    Universities today teach students to use IDE, and solely work with that. When I reach out to people who have stopped contributing, the lack of a modern IDE is a major response.

    The new “gbuildtojson” that Bjoern has made, is a step in the right direction, but we are still far from a goal, where a contributor clone the repo and open his preferred IDE to start working.

    Investing in: - removing cygwin in Windows - making a auto installer (lode++) - complete gbuildtojson and xx-ide-integration to include prepared solutions for popular IDE (msc, Xcode, eclipse)

    would help a lot to keep the contributors contributing, and thus help grow the community.

    This project has a very wide scope.

    More CI hardware to get quicker build-times

    CostEstimate: Eur 20k Contact: Noel

    • consider cloud hardware cost; scale on-demand ? (Bjoern)
    • Norbert considers this to be mostly a problem of stopping builds jamming existing hardware

    post Macs to people

    CostEstimate: Eur 9k Contact: Michael

    Lots of core contributions don't have Macs - this makes it really hard to ensure that work which has a cross-platform impact works well on Mac, and (more to the point) that bugs and regressions that affect Macs are addressed. Remote Macs have problems of maintenance, and remote desktop protocols don't work at all well: broken keyboard mappings, and corrupted screens. To solve this we should loan Macs on-site with key developers who make some commitment to look at Mac issues using this as/when they arise. A list of such developers is:


    • Khaled Hosney - low-level text rendering.
    • Michael Meeks - VCL pieces.
    • Jan Holesovsky - various fixing
    • Jan Marek - VCL / main-loop testing
    • Thorsten Behrens - Office Mac to share with Bubli, Armin, Oliver etc.
    • Samuel Mehrbrodt
    • Caolan McNamara - all over the code
    • Michael Stahl - all manner of filters & fixes
    • ... others ? ...

    Note 1: Dont we even still have a infra mac idling in the lab?

    Note 2: For occasional users, how about shared machine with Chicken Of The VNC Access?

    Preconfigured Cloud VMs for Newcomers

    CostEstimate: Eur 10k Contact: Bjoern

    Have ready-to-go VM with a preinstalled Dev Env including IDE integration with debugging and code completion for the most popular platform (Windows=MSVS, Linux=kdevelop, macOS=XCode) and in the best case a prebuild LibreOffice. Those imagine should be able to be spun up by the dev-mentor at a minutes notice for newcomers to submit their first 3-5 patches. This is asking for budget for the hosting (e.g. cloudonmac or cloudshare) only, I have hope that both volunteers and the dev-mentor will be able to script the setup of these base images.

    see also: https://wiki.documentfoundation.org/Hackfests/VMs/Using_a_VM

    (This item likely can be folded into the infra budget in the end, which is fine: Then we wouldnt need to vote on this separately.)

    finishing online help -> make it actually online

    CostEstimate: 6 weeks Contact: Kendy

    • finishing the XHP generating JS, sort out translations,
    • ensure it works off-line with searching; and finally killing help viewer.
    • ideally also online editor (that would upload patches to gerrit)
    • tender submitted for BoD (Olivier)

    Collaboration Dictionary Project

    CostEstimate: 8 weeks Contact: DennisRoczek

    Collaboration Dictionary Project was discussed at I10N: a platform for creating and updating dictionaries and automatically pack extensions from word lists (and upload them to the web pages).

    Accessibility improvements

    CostEstimate: 2 weeks Contact: Michael

    Add tool to find and flag new glade widgets that are added without a11y markup - ie. any new label without a relation for the widget it relates to - should cause a compile / tinderbox warning - except in the case that it is used as a hidden string placeholder to avoid the resource files. Catching the common cases, and black-list all the existing dialogs and/or widgets without these. This should avoid future a11y regressions in the markup that we can scan.

    Blocking features

    SmartArt implementation

    CostEstimate: 16 weeks Contact: Thorsten

    Our existing smart-art import uses the fallback-stream in OOX files, and gives us only the draw shapes that are imported - thus loosing the original layout. In older file versions we don't have the cached shapes - so can render nothing. We really would like schema driven diagram layout as a core feature as well. This should be interoperable with OOX, and have suitable extensions for ODF. It should layout interoperably, and allow editing of the underlying data, and selection of a schema.


    HSQLDB binary format import

    CostEstimate: 6 weeks Contact: Michael

    In order to remove the legacy Java / HSQLDB database completely, and move fully to Firebird in 5.4 it is necessary to be able to import old document data with high fidelity from the HSQLDB binary file format which we have used (for performance) inside so many of our existing ODB files. This task involves reading the existing (reasonably simple) Java serialization code, and writing an (import only) filter to import this data safely into base.

    Improved built-in scripting debugging

    CostEstimate: 10 weeks Contact: Michael

    XRay-alike, to be built-in as native code, with better core hooks and usability etc. Modern browsers come with significant, helpful built-in JS developer tools, and we are far behind this in terms of ease-of-use for development of extensions and (more critically) unit tests using the UNO APIs.

    Porting the XRay functionality from StarBasic to C++, and including it - along with providing a cleaner, browser-like UI at the bottom of the screen would be ideal.

    In particular - the ability to select an item in an arbitrary document eg. an image, or a shape - and get its UNO peer, and be able to inspect attributes on it would be excessively helpful.

    Similarly logging UNO events emitted by the core, and allowing inspection of their details in a console-alike.

    finish the XServiceDocumenter to allow UNO user to learn about core implementations

    CostEstimate: 2 weeks Contact: Bjoern

    Finish the com.sun.star.script.XServiceDocumenter/theServiceDocumenter to allow UNO and StarBasic users to use a call to xDocumenter.showCoreDocs() to get directly to docs.libreoffice.org showing the C++ classes implementing a specific UNO service. This should smooth the migration of interested people to move from UNO/extension code to core code and scratch their itch.

    Note that the showServiceDocs()/showInterfaceDocs() methods of the service documenter (that bring you to the abstract UNO docs on api.libreoffice.org) are already implemented, so not in scope for this task.

    (the XServiceDocumenter.showServiceDocs()/showInterfaceDocs() could also be a base for "Improved scripting debugging". Possibly merge these?)

    see: https://www.youtube.com/watch?v=WBNG6bVZPzw for details

    Improve the experience of the SDK

    CostEstimate: 4 weeks Contact: Bjoern

    UNO API changes subscription/feedback channel

    CostEstimate: 1 week Contact: Bjoern

    The simplest solution (KISS for a start) would be e.g.:

    • an archived mailing list (uno-changes@ ...)
    • an bot that puts every API change on that list, cc'ing the author of the change,
      • someone replying to the API change also reaches the author of the change
    • bot that filters for the API change discussions and put notifications and RSS feeds on api.libreoffice.org

    UNO API discussion/sniplet dump

    CostEstimate: 4 weeks Contact: Bjoern

    PHP is a horrible language, still it succeeded in capturing a huge amount of the market. I assume, the "user contributed notes" on their API documentation played a major part in that (see e.g.: http://php.net/manual/en/function.strlen.php). This proposal is to add a similar user-contributed code examples directly to a specific part of the UNO API documentation at api.libreoffice.org. The discussion software should have a low barrier to get into (OpenID?) and support e.g. code highlighting. Askbot might be a candidate.

    This is pretty much a superset of the "UNO API changes subscription/feedback channel" propoal: those bots and RSS feeds could use the discussion forum.

    UX problem areas

    User Metrics

    CostEstimate: 12 weeks Contact: Kendy

    Re-run the User Metrics tender as before.

    [ https://blog.documentfoundation.org/blog/2015/02/24/tender-to-develop-and-incorporate-usability-metrics-collection-for-libreoffice-201502-02/ ]

    Redesign of chart wizard UI

    CostEstimate: 6 weeks Contact: Bubli

    Improving printing UI

    CostEstimate: 20 weeks Contact: Bubli

    Already Resourced / Abandoned

    Automatic Detection and Bisection of Regressions

    CostEstimate: in-progess Contact: Luke Document and improve the https://github.com/milossramek/office-interoperability-tools tools described at https://www.youtube.com/watch?v=cfzMP4oTZwo for the QA team. Add into LO QA framework. Have an end goal of reaching a state where it could run automatic weekly reports against a pool of test documents. Xisco: I'm already working on it: https://github.com/x1sc0/office-interoperability-tools

    ongoing

    could we have a 'tips' scheme

    CostEstimate: ? Contact: Heiko

    • KDE side, use pay-pal only https://www.kde.org/fundraisers/yearend2016/
    • sounds like re-inventing freedom sponsors (Bubli)
    • if this happens - do it outside the foundation to avoid issues (Norbert)
    • like barnstars but with a financial 'tip' - is the idea.

    abandoned - Norbert (TDF treasurer) is not a fan; so lets do it outside TDF funding.

    patch update code

    CostEstimate: unclear yet / just for Mac? Contact: Markus

    • allow pushing patches, lots of details to sort out
    • have a FOSDEM talk for this in the dev-room
    • talk to releng & devs there @ the hack-fest.

    => not sure it will work out as a tender; already 70% done.

    • Windows & Linux ~done; no Mac so not tested

    defer - until after FOSDEM.

    Proposals missing detailed descriptions for tenders in budget 2017

    Better integration of extensions (Heiko)

    Design-team page of topics for GSOC (Heiko)

    Proposals for tenders in budget 2017 that are likely better suited as a GSOC

    Subpixel and stable glyph positioning

    Contact: Khaled Fixing the bad text spacing we have due to lack of subpixel positioning. All graphics system we use (except GDI, but this is dying) support subpixel positioning but we make no use of it as all our glyphs positions are stored using integers (not to mention the rounding errors all over the place). This results in very bad text spacing and it gets worse on hidpi screens. Thinking about this more, I think it can be a GSoC project.

    Add Impress remote capabilities for simple ad-hoc fingerpainting ad-hoc sketches

    Contact: Bjoern Likely better suited to GSOC -- adding it here as we collect budget extensions. The idea is basically to allow to do simple ad-hoc sketches on a tablet with the impress remote, e.g. to answer an audience question on the spot. Allowing the tablet to use the projector as a modern chalkboard.

    Make Calc/Base/Chart handle small set of geographic data

    Allow Calc/Base/Chart to handle regional subsets of geographic data, e.g. import/edit/draw a small region of OpenStreetMap data.

    see also: https://blog.documentfoundation.org/blog/2016/09/19/prototypefund-an-opportunity-for-german-freelance-developers/

    Retrieved from "https://wiki.documentfoundation.org/index.php?title=Development/Budget2017&oldid=643650"