Making CryptPad CSS 3 times faster, by loading it twice

In CryptPad development, we have always tried to push the limits of the technology. As you might know, we don’t minify any of our javascript code and we have no build system, yet CryptPad is still faster than many similar projects which do. In recent profiling, we determined that the biggest cause of slow loading was compilng of less stylesheets.

All of the styles for CryptPad are written in Less CSS templating language and because we don’t have a build system, when you load a page on CryptPad, it downloads the less compiler and runs it in your browser. When the less has been compiled the first time, it is cached in the browser’s localStorage so that it doesn’t need to be compiled again (until next release). Unfortunately, the way we structured CryptPad Less code led to this taking a long time.

CSS by its nature is very much like object inheritence, a design pattern popular in the 1990s which has since been discredited. In a bid to keep our styles under control, we decided to make heavy use of Less mixins. The idea was that we didn’t want our CSS code to “speak until it was spoken to”.

Something like the following would be very problematic if it were dumped on the global scope, but it is never output until it is called:

1
2
3
4
5
.example_header() {
a {
color: red;
}
}

So then the code which uses it can invoke it only in the exact place where it ought to be used.

1
2
3
4
5
6
7
8
@include "example.less";
.cp-app-pad {
.cp-padheader {
.cp-padheader-left {
.example_header();
}
}
}

If you’re a CSS purist, you’re probably pulling your hair out now, because the right way is to use html classes. The thing is, we do, but because CryptPad is made up of many different pieces of open source software, we cannot control all of the HTML and sometimes things aren’t so simple. Using mixins gave us an extra layer of safety and allowed us to write CSS feeling quite confident that it would not end up changing things where it wasn’t supposed to.

Cascading works great, until you include one CSS file that was written by this guy.

Parameterized mixins

An excellent feature of Less is parameterized mixins, that is, templates with arguments. One of our biggest templates is called .toolbar_main() and this builds the toolbar at the top of CryptPad as well as the user-list on the lefthand side. In order to make the customizable colors with the same HTML structure, we opted to use a parameterized mixin like this:

1
2
3
4
5
6
7
8
9
.toolbar_main (
@color: @colortheme_default-color, // Color of the text for the toolbar
@bg-color: @colortheme_default-bg, // color of the toolbar background
@warn-color: @colortheme_default-warn, // color of the warning text in the toolbar
@barWidth: 600px // width of the toolbar
) {
/// a lot of code here, using colors based on the parameters,
/// but lightened or darkened using less functions.
};

Then each app would call .toolbar_main() with its own color parameters, and get a nice toolbar, customized to that app’s color theme. Again, this is not the only way to do it, but having the ability to generate CSS with highly specific color rules proved to be extremely useful for overriding leaked styles coming from the software that we integrate.

State explosion

What we didn’t think about at that time was the effect that the proliferation of CryptPad Apps would have on the amount of CSS being generated. We started with just a few styles and just a few apps to apply those styles on.

But as we added more and more CryptPad apps, the same CSS was being generated and applied over and over…

Then as our styles become more complex, the CSS which was being copy/pasted by less compiler became bigger and bigger

The total of all our less code in the entire project was only 235k, and it was compiling to over 1.3 megabytes of CSS. We cache the compiled CSS by placing it in localStorage, but still, every time a new version of CryptPad was released, the CSS needed to be recompiled and this was dominating the loading time.

Building a Linker for CSS

If you have experience with C/C++, you might recognize this problem. It is as if there was no linker and the only way to reuse code was to use preprocessor #include over and over again.

Since by this point, we had a significant amount of Less which was designed this way, rewriting it was not an option, so we started looking for ways to link it rather than copy/pasting it over and over again. Fortunately most of the bigger mixins only applied rules to specific classes, so moving them up to the root level would not cause trouble, though to be safe, we wanted to only load the styles that were necessary.

If we would indicate to the javascript code which loaded the Less that a particular Less file was needed, it could be compiled, cached and included separately, and thus it could be reused across apps. In order to keep the Less API as close to the same as possible, we decided to put that indication inside of the .<filename>_main() mixin.

So .dropdown_main() went from this:

1
2
3
.dropdown_main () {
// all the code here
}

To this:

1
2
3
4
5
6
.dropdown_main () {
--LessLoader_require: LessLoader_currentFile();
};
& {
// all the code here
}

Two important things to note: firstly LessLoader_currentFile() is a function which we created (as the name implies, it’s defined in LessLoader.js), it simply expands to the current function name. Secondly, when a less file is included with the reference flag (e.g. @include (reference) "./dropdown.less";) the content is not output but the mixins are made available, so moving the code down to the bottom of dropdown.less would cause it not to end up in the compilation of app-pad.less.

The resulting CSS from this contains something like this:

1
--LessLoader_require: "/customize/src/less/include/dropdown.less";

LessLoader would then simply scan for --LessLoader_require: and trigger loading of those files, which are still parsed by Less, but are the same for every CryptPad app.

Parameters with CSS variables

In this example, I intentionally left out the parameterized mixins. Solving this was slightly more complicated and in order to do it, we made use of a reasonably new feature in web browsers: CSS variables.

Today, one can write in CSS the following:

1
2
3
4
5
6
7
8
:root {
--color-should-be: brown;
}

// potentially much later...
.element {
color: var(--color-should-be);
}

and the element text will be brown. Discovering this was a breakthrough because it meant that the arguments could be turned into variables in app-pad.less and then made use of in toolbar.less. However, there are limitations to what you can do with CSS variables. For example, this doesn’t work:

1
2
3
4
5
6
7
8
9
10
:root {
--hack-boolean: 1;
}

// later on...

@media screen and (max-width: calc(var(--hack-boolean) * 100000000)) {
// HAHAHA I MADE AN IF STATEMENT
:root { --lets-define-another-variable: "lol"; }
}
If this worked, I’d probably be using it


Scoped CSS variables

What does work, however, is specifying different values of the same variable at different scopes, so this does work:

1
2
3
4
5
6
7
8
.my-button { --button-color: red; }
.my-popup-window .my-button { --button-color: blue; }

// later on...

.my-button {
background-color: var(--button-color);
}

But possible uses/abuses of this feature were not investigated.

Making it work

Following the general principle of keeping the variable definition close to the usage, we put the variable definitions inside of the .<filename>_main() mixin and the usages below in the same file. In order to avoid namespace collisons, we prefixed all variables with the name of the file. A simplified version of avatar.less looks like this:

1
2
3
4
5
6
7
8
9
.avatar_main(@width) {
--LessLoader_require: LessLoader_currentFile();
--avatar-width: @width;
}
& {
&.cp-avatar {
...
.cp-avatar-default, media-tag {
width: var(--avatar-width);

In some cases, we needed to introduce additional variables because of the use of Less functions such as lighten() and darken(), which obviously cannot work on CSS variables. So we used the following pattern in many places:

1
2
3
4
5
6
7
8
9
10
.help_main (@color, @bg-color) {
--LessLoader_require: LessLoader_currentFile();
@help-bg-color-l15: lighten(@bg-color, 15%);
@help-text-color: contrast(@help-bg-color-l15, #fff, #000); //@color;
@help-link-color: contrast(@help-bg-color-l15, lighten(spin(@bg-color, 180), 10%), darken(spin(@bg-color, 180), 10%));

--help-bg-color-l15: @help-bg-color-l15;
--help-text-color: @help-text-color;
--help-link-color: @help-link-color;
};

Results

After carefully planning and studying solutions, we managed a fairly non-invasive refactoring of the styles which took Less compile time down from almost 3 seconds to around 900ms. For simplistic pages like the front page, the number dropped to around 200ms.

The key result is that a person who has never seen CryptPad before will see the main page right away instead of waiting 3 or more seconds to compile all the less for the entire project.

But wait, what about Internet Explorer ?

This question is the bane of many web developers’ existance. In this case, the problem is that Internet Explorer has no CSS variables. Last week we changed CryptPad so that when you use it, your browser will let us know if it doesn’t support CSS variables. This is done using the feedback mechanism which is an opt-out collection of information such as how often particular features are used and whether certain things are supported by the browsers of people using CryptPad.

What we found is that in the past week, we saw about 50 unique users who are running browsers which don’t support CSS variables. With our approximately 4500 unique users per week, this is a little over 1%.

Making an acceptable fallback

The One Percent jokes asside, it’s hard to justify making CryptPad 300% slower for everyone who tries it for the first time, just to satisfy about 1% of the userbase. But at the same time, it’s sad to drop support for a browser which at the current moment does work with CryptPad.

The solution we devised was to specify default values and then override them.

For example:

1
2
3
4
5
6
7
8
.cp-markdown-toolbar {
...
button {
// IE sees this (variable compiled by less)
color: @toolbar-color;

// everyone else sees this
color: var(--toolbar-color);

The only question remaining was how to specify the defaults in a way that was simple for us when we worked on the less files. Most of our .<filemane>_main() parameterized mixins already had default values in case they were called without parameters, so we already knew what sane defaults would be. What we decided to do was create a new mixin called .<filename>_vars(), which would assign a set of Less variables based on the arguments.

The final result looked like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
.help_vars (
@color: @colortheme_default-color,
@bg-color: @colortheme_default-bg
) {
@help-bg-color-l15: lighten(@bg-color, 15%);
@help-text-color: contrast(@help-bg-color-l15, #fff, #000); //@color;
@help-link-color: contrast(@help-bg-color-l15, lighten(spin(@bg-color, 180), 10%), darken(spin(@bg-color, 180), 10%));
}
.help_main (@color, @bg-color) {
--LessLoader_require: LessLoader_currentFile();
.help_vars(@color, @bg-color);
--help-bg-color-l15: @help-bg-color-l15;
--help-text-color: @help-text-color;
--help-link-color: @help-link-color;
};
& {
.help_vars();
.cp-help-container {

position: relative;
background-color: @help-bg-color-l15;
background-color: var(--help-bg-color-l15);

Going through it step-by-step, the .help_vars() mixin takes parameters but it defines default values, then it creates some Less variables which are accessed after it is used. .help_main() calls .help_vars() and passes it arguments, then takes it’s results and assigns them to CSS variables. Then the main block of the Less file also calls .help_vars() but without any arguments, so the defaults are used. Then in the main block, each usage of a CSS variable also has the usage of the less variable. The less variable provides the default value, the one which IE will see, and then the CSS variable provides the specified value, the one which will be different per CryptPad application.

There you have it

Every Less file now gets loaded twice, first it is loaded by the @include (reference) call, where it is parsed in order to expose the .<filename>_main() mixin, then it is loaded a second time as an independent file. Importantly, however, once it is compiled it is stored into localStorage and the toolbar CSS which is compiled when you go to the Rich Text app is the exact same CSS which will be used when you go to the Code/Markdown app. This feature is available in the staging branch of the CryptPad project and will be on CryptPad.fr in the next release.

We could make it even faster, by separating the main content of the Less files from the .<filename>_vars() and .<filename>_main() mixins, essentially making header files, in C/C++ parlance. And if we find in future profiling that less compiling remains a signficant performance penalty, there’s a good chance that we will.

CryptPad Roadmap

CryptPad was started as an experimental platform as part of the OpenPaaS::NG research project. Since then it has developed into a suite of editors for many different types of documents, all without ever leaking the content that you edit to the server or the server operators. Now, in order to keep the project moving, we need your help.

We have created a roadmap of the features which we would like to develop over the next year and we’re hoping to raise 60,000€ to finance it. Fortunately, XWiki SAS is willing to match your donations euro-for-euro. XWiki SAS is a company with open source at its heart and it was the company where CryptPad was first envisioned. XWiki SAS normally sells days of development time at about 1,000€ each, but since this project is dear to our hearts, we are also discounting the structure costs (things like the office and business taxes) for this project. We believe in CryptPad, but in order to succeed, we need you to believe in it too.

The following is a 48 week roadmap which will fund one developer to work on CryptPad. You can donate to this roadmap on https://opencollective.com/cryptpad or on https://accounts.cryptpad.fr/#/donate, if you would like to make a donation by wire transfer or by cryptocurrency, please get in touch with us at sales@cryptpad.fr and if you are involved in EU research projects and would like to work with us and this cutting edge Privacy Enhancing Technology, please contact research@xwiki.com.

Goal 1 - Spreadsheets & Office Documents

CryptPad is excellent for quick editing of meeting notes and plans, and it is able to store and view images and PDF files, but it does not allow editing of more complex types of files such as spreadsheets and word documents. Fortunately there is an open source web-based editor called OnlyOffice which allows editing these complex document types, but it is a highly complex piece of software and we will need time and effort to integrate it into CryptPad. Even after it is fully integrated, the conversion of xlsx files to OnlyOffice’s internal format will still need to be carried out be specialized software either on the user’s computer, or on a cloud service (not Zero-Knowledge!). However, we think that editing of spreadsheets and other documents in OnlyOffice is achievable without leaking any of the content back to the CryptPad server, so your spreadsheets will remain a secret.

  • Spreadsheets, documents and presentations in CryptPad
  • Realtime collaborative editing of office documents
  • No import/export for now, this will come later
  • No embedding of images from CryptPad until later

We expect office file editing will take 30 days, a cost of €7,500

Goal 2 - Comments on Rich Text Pads

Many people use a workflow which includes the use of comments in a document. This is already standard in Google Docs and we need to be able to support it in CryptPad, in order to add comments we will need to add some HTML into the document without causing any problems for CKEditor (our rich text editor) and without causing it to be treated as part of the pad and synced over the wire.

  • In the rich text pad, you will be able to add a comment by selecting some text, right clicking it and clicking “add comment”
  • Comments will be shown in the right margin and associated with the text which highlighted them
  • Comments will be able to be replied to, or “resolved” (they will no longer be shown, but still be visible in history of the document)
  • If you delete the text with which a comment was associated, the comment will automatically be resolved

We expect Comments on Rich Text Pads to take about 10 days of development, a cost of €2,500

Goal 3 - Contacts and Messaging

To be able to easily share documents and get the attention of a person, we would like to implement Contacts and Messaging in CryptPad, currently we have a rudimentry implementation which allows chatting with contacts, but the contact request feature needs to be stabilized with a better process for contact requests, better integration with pad-sharing menu and more clear notifications. We would also like to add a small chat window in each document so that everyone working on that document can chat about it. Finally, we would like to add the ability to at-reference someone when writing a comment on a pad, which will cause a message to be sent to them.

  • The share menu will include a list of contacts with whom you can share the pad
  • You will be able to email somebody a link which will invite them to become a contact of yours
  • Messages (pads shared with you, comments referencing you) will be shown in the upper-right when you come to CryptPad
  • When writing a comment (Goal 2), you will be able to at-reference a contact (with auto-complete) and they will be notified
  • Replying to (and resolving) a comment will also cause the author of the comment and anyone who replied to be notified

We expect Contacts and Messaging to take about 20 days of development, a cost of €5,000

Goal 4 - Shared Drive

When planning a significant project, often there is the need to have many pads for the different aspects of the project. Currently all of the collaborators on the project must share all of the pad links with each other and they must each organize them in their personal drives. Personal drive organization is a nice feature because it allows each person to organize their work the way that makes the most sense to them, but sometimes it is more advantageous to have a folder in everybody’s drive which contains pads that are shared between all of them.

Shared folders are complicated to implement. Unlike your personal drive which is just one realtime object (essentially it’s a pad), each shared folder must be a separate realtime object. Also, like a user account, a shared folder must be able to own pads, otherwise pads in that folder which are not in anyone’s personal drive risk being deleted by the server as per our expiration policy.

  • You will be able to create a shared drive, just like you create a pad
  • In your drive, you will be able to explore inside of the shared drive and organize it just like a folder
  • You will be able to share the link to the Shared Drive with other people, when they click the link, it will import the shared drive into their personal drive

We expect Shared Drive to take about 20 days of development, a cost of €5,000

Goal 5 - Federated Messaging

There are about 150 CryptPad instances in existence, operated by people and organizations who want the privacy which CryptPad offers and also the additional security guarantee, decentralization and customization of their own instance. We want to support this usage while still allowing people on one instance to chat and message people on another instance. We want to support the ability to add contacts (Goal 3) with people registered on a different instance. You will then be able to at-reference them in a comment or share a pad with them using the share menu as you would with a person using the same instance as you.

  • When you are on a pad or shared folder on somebody else’s instance, you will be able to import it to your instance
  • Clicking a link to become a contact will work, even if you are registered on a different instance
  • Once you have made a contact, all features in Contacts and Messaging will function across instances

We expect Federated Messaging to take about 30 days of development, a cost of €7,500

Goal 6 - Better permissions

Initially, all pads were “open” pads, meaning they were accessible to anyone and there was no owner. Open pads are deleted automatically when they have not been touched in 3 months, unless a registered user has a reference to them. Once we added the ability to have registered users, we needed to make sure that users would not have the pads in their drive disappear off the server, so we implemented what we call pinning. Pinning is a way of claiming that a pad is important to you and that it should not be deleted. However, this left a problem, still pads had no owner, meaning nobody had the right to delete them. Once a pad was created, there was no way to delete it which was a serious problem for pads contining confidential information.

In order to allow pads to be deleted, we implemented owned pads, and the splash screen registered users now see when they create a new pad. So now you can delete it from the server when you are done with it. However, two problems remain:

  1. The owner of a pad is the person who created it and there is no way to add owners or share ownership
  2. There is no way to revoke access once it has been given

In order to solve #1, we will add the ability to share ownership with other people. For solving #2 in a Zero-Knowledge compatible way, we will need to introduce the concept of redirects, links which re-direct to the actual ID and key of the pad. Revoking access will be a matter of changing the pad’s ID and then updating all of the redirects which are not to be revoked. This will allow the creation of pads which can only be viewed by registered users who are invited. Even view-only links to a pad will be able to be revoked by the owners of that pad.

  • Pads which are only accessible by people invited, or members of a group
  • Grant and revoke access
  • Create new links to a pad, and revoke them

We expect better permissions will take about 40 days of development, a cost of €10,000

Goal 7 - Color-by-author

After something is typed into a pad, the person who typed cannot be determined, we would like to change this and allow the etherpad-like behavior of shading the text differently based on who typed it. This will require getting the author from each patch in the stream of changes to the pad and making it available to the pad structure. In CryptPad there is a special message type which is called a checkpoint, it replaces the entire pad so that when a new person arrives in the pad, they don’t need to download more history than the previous 2 checkpoints. We will need to take special care to preserve the editor of each piece of the document when we create checkpoints, finally we will need to be able to shade the text based on it’s author, without either messing up CKEditor or causing the shading to be sent over the wire as if it were part of the pad.

  • When an option is selected, text in the rich text app will be shaded with a color based on the author of that text
  • The user list will show users who have joined and then left again, along with the color of their changes
  • Right-clicking on the text will show the name of the person who wrote it (including whether they are one of your contacts)

We expect Color-by-author to take about 20 days of development, a cost of €5,000

Goal 8 - Offline and Suggested Edits

Like color-by-author, Offline and Suggested Edits is a reasonably complicated feature to implement. When working on a complex document, it is nice to be able to propose a change without actually changing the document, or make a change while disconnected from the internet and then have someone (maybe you) merge it later. However, this is complicated because when it comes time to merge that change, the document may have significantly changed, necessitating a smart merge.

Rich text pads in CryptPad are stored in a quazi-html representation so patching and merging is anything but simple. The operational transformation done normally on pads takes advantage of the fact that divergences are small and infrequent so merging usually works ok. This feature will require development of a smart merging algorithm for the rich text app as well as for any other app for which one would like to have suggested edits and offline.

  • Suggestion mode, your change is shown shaded and can be accepted or rejected, it will be attached to a comment so you can explain it
  • When offline, you are automatically put into suggestion mode instead of read-only mode

We expect Offline and Suggested Edits to take 40 days of development, a cost of €10,000

Goal 9 - All the little things

Even though it doesn’t get a lot of press, a significant amount of time is spent just fixing bugs, handling pull requests and doing releases. We want to keep CryptPad a community project and so we want to make sure that when someone makes a pull-request to the project, that request can be properly handled. Also, because CryptPad makes such use of modern HTML5 features, new releases of Chrome and Firefox typically introduce new bugs in CryptPad. There is rarely enough time to handle all of the issues which crop up, but we need to allocate time to handle some of them. This roadmap allocates the time of one person for 48 weeks, which means 24 releases, we need at least 1 day per release just to perform the release and write the release notes and fix miscellanious issues which appear.

  • Keep on fixing bugs in CryptPad
  • Handle pull requests and make releases

We expect all the little things to take 30 days of development, a cost of €7,500

CryptPad #ZeroKnowledge Free Software needs funding

Version française ? 🇫🇷

On October 31, 2014 the CryptPad project was first published, back then it was nothing more than a simple rich text pad and a horribly ugly front page. Since then the project has blossemed with financing from XWiki SAS and the OpenPaaS::NG research project.

Now we are turning a corner, the OpenPaaS::NG project will end in April of 2019 and in order to keep improving, CryptPad will need new sources of funding.

Until now, the vast majority of CryptPad code has been developed by XWiki SAS and with the future of research financing in question, we want to be completely clear about our intentions. We don’t want to take financing that is incompatible with the open spirit of CryptPad so while we will look for research projects, subscriptions and donations, we want to maintain the community spirit of CryptPad.

  1. We will continue to host https://cryptpad.fr/: We want to thank everyone who has subscribed to paid accounts, with your support (currently 1,5K/year) this server pays for itself and we consider this a valuable public service which should continue.
  2. XWiki SAS will continue to fund the team through the OpenPAAS project till the end of the project. Following this our intention is to try to keep at least 1 developer active on the project (50,000€ per year) using alternative funding or through new research projects. If we can’t find a research project we’ll evaluate our capacity to keep it alive based on the subscriptions and contributions we have received.
  3. We will begin making the CryptPad finances public: All of the money that comes in and goes out of the CryptPad project, including the money paid by subscriptions to https://cryptpad.fr/ will be published so that you can see how the project earns and spend money, and be convinced that it’s worth supporting us. You will be able to follow the project’s finances by going to CryptPad on OpenCollective

Currently, the funding from our subscriptions and donations is not enough to finance even one developer, nevermind the present team of two. We need your help to grow this revenue through April 2019 and show that this can take over funding even if we don’t find a research project before then. All revenue received till April 2019 will be used to fund development after April 2019.

What we would like to do

So far, we have not been very transparent about our roadmap, we have a tech tree which shows in simple terms what features we would like to have and what technologies need to be developed in order to get those features working.

In the medium term, we would like to see CryptPad evolve into a generic platform with installable apps, a cryptographically enforced access control system, and federation with PGP-compatible messaging.

  • Installable Apps: There are not a lot of differences between the rich text pad, the code pad, the slide deck and the kanban board. They are all layered on top of the same CryptPad base infrastructure, but still it is not possible to create a new one without changing a few things in the CryptPad core. We would like to change this so that a new app can be added without changing anything else.
  • Cryptographically enforced Access Controls: Initially CryptPad had a very simple access control system, you share the link and that shares the pad. We added the ability to publish pads via read-only links and to assign a password to a pad, but we still don’t have the ability to share a pad with a group of people, or importantly, revoke access once it has been given. Traditional access controls are simple because the server is trusted to give and revoke access, however with CryptPad the server doesn’t have access to begin with so it must be done cryptographically. In the context of federation, cryptographic access controls are even more important since servers in the network can be run by anybody.
  • Federation and Messaging: The weakness in CryptPad security has always been securely sharing the pad link. Today there are about 150 CryptPad instances installed around the world, and we would like to allow people on different instances to share pads and send eachother messages. Furthermore, since we already have client-side encryption, we could easily extend messaging to support PGP for sharing of pad links and messages.

Fundamentally, our goal is as it always has been, to promote Zero Knowledge and an alternative to the Google Docs / Office365 hegemony on cloud office technology. However, this is not something we can do alone, we will need your help to move it forward.

How you can help

You can help CryptPad in a number of ways, if you’re a programmer then you can contribute code, if you’re a philanthropist and believe in these ideas then you can finance the roadmap or finance just particular features. What everyone can do is use CryptPad and spread the word and show people that it is possible to collaboratively edit documents without giving all the data to the server admins.

  • Contribute code: If you are a programmer and you are using CryptPad, help make it better, talk to us about what you would like to do with CryptPad and we will do our best to find a way that your code can be integrated.
  • Take a subscription: Every subscription helps bring a little more money into the project and this will be re-invested to make the CryptPad project better. Every subscription we get makes us believe more in the project we do and the ability to make it financially sustainable. Subscribe.
  • Sponsor a feature: This is one of the best ways to make sure CryptPad will improve because you can both help the project and help guide the project roadmap at the same time.
  • Get support: If you’ve installed your own CryptPad in a business setting, you can get support for your installation and also help foster development of the project.
  • Just donate: If you don’t know what to sponsor, if you don’t need a subscription, or if you want to sponsor more than the value of a subscription, just donate ! Even if it is not a lot of money, this is important to us as it allows to prove that the project matters. Donate on opencollective.com
  • Get us in a funded research project: If you are a research organization or have experience with getting European research funding, particularly in a security oriented project, get us on board to participate in the project. This will allow us to fund the project.
  • Spread the word: The more CryptPad is used and the more it’s known, the more we can convince potential financers to fund the project, this is also proves that people are interested in it. Tweet about CryptPad

To see information about our budget and goals and to make a donation, check out CryptPad on OpenCollective.

Come have a chat

We’re on Twitter and Mastodon and we have a CryptPad chat room on Matrix. Come talk with us and participate to help with the project.

If you are a Web or Research professional, are intereted in our objectives and have experience in research, come talk to us as we’re interested in hiring a person to help us win and execute research projects and guide CryptPad to make it functionally and economically successful.

Faster loads with SharedWorker & ServiceWorker

When CryptPad was first created, the only thing to load was the CryptPad code itself and the pad which you were editing. Recently edited pads were remembered in the browser’s localStorage which was not portable between computers but allowed some recent history to be kept.

However, we wanted to allow people to login and manage their pads on all of their devices so we created the CryptDrive. Anyone who has used CryptPad but never logged in is encouraged to register and login (it’s free) and check out their drive. CryptDrive is basically just a realtime pad containing a JSON structure with links to all of your pads as well as their titles and other information. When you update the title of a pad, it changes the pad itself but it also changes your drive so that you can see the title of the pad in your drive. This is of course not perfect because if someone else changes the title of a pad, your drive will not be updated until you look at the pad again, but doing everything with the server completely blind to the content isn’t easy, and this works reasonably well.

However, CryptDrive causes an additional delay when loading CryptPad because whenever you load a document, you are actually loading two realtime instances. Since the drive is loaded over and over for every pad you view, it was obvious to us that we could make it more efficient using communication inside the browser.

First idea: Messages between tabs

Tabs (or windows) in a browser which are on the same website are able to communicate using localStorage and so it seemed like a good solution to just have one tab claim the role of managing the drive and then when another tab is opened, it would message the first. However, this seemingly easy solution becomes a nightmare when you consider what happens when that tab is closed. The drive that everyone is relying on goes away and all of the other tabs are without a drive so they need to flip a coin to decide which tab should become the keeper of the drive, then that tab needs to download the drive before it can service events, all the while any of the buttons which affect the drive (for instance deleting the pad) cannot possibly function. This idea was soon scrapped…

Enter ServiceWorker

Recently, the HTML5 working group created a new standard called ServiceWorker which for someone making a webapp seems like a dream come true. ServiceWorkers:

  • Are side-processes which are created one-per-website and live in the background.
  • Can intercept HTTP requests from your main javascript (excellent for caching!).
  • Is suspended when the last tab is closed, and re-launched when the user returns.
  • Are stored in suspended state even when the browser is turned off.
  • Supported by every modern browser.

When the website loads a worker, if the worker is already running it will not load and will instead defer to the existing worker. Communicating between workers and tabs which are on the website is possible via a postMessage() API and then the ServiceWorker can postMessage() which will reach all tabs that are navigated to the website.

One limitation of this design is versioning. Because the ServiceWorker would stay alive potentially forever, we needed to identify a way to upgrade it if a new version of CryptPad is released. This is quite important for CryptPad as version mismatches can lead to catastrophic conflicts between different browsers working on the same document.

Though updating is non-trivial, we were able to solve it by sending version messages between different components of CryptPad and informing them whether they need to update (or even if they can optionally update). Since this seemed like a solvable problem, we tried creating an experimental implementatiton of ServiceWorkers in CryptPad, and then the fun started…

ServiceWorkers in Firefox

Since Firefox 48, Firefox has begun following Chrome’s model and running different processes for rendering the different tabs in the browser. However, this isolation has a side-effect that when you attempt to launch a ServiceWorker, it may launch even though one already exists for the same website, because the other one exists in a different tab which happens to be operating in a different process. However, the postMessage() requests from tabs go out to all of the ServiceWorkers so this bug can be worked around.

Unfortunately we encountered some more issues with Firefox which we found not worth debugging. Our CryptDrive is a Javascript object which is represented in the encrypted realtime document as JSON and there is an ES6 Proxy object in order to allow every change to the drive to be propagated to the underlying realtime object. In Firefox the proxy did not work when used in a ServiceWorker and at times our ServiceWorker simply stopped running.

Reporting bugs in browsers

One might think these are great opportunities to report issues with Firefox, and when we find an issue which we think will affect lots of people especially if it is a regression, we don’t hesitate to report it, but usually it is not clear how to reproduce the issue, whether there might be some error in our usage of the API which is being smoothed over by Chrome, or whether the component being reported on is a priority, in that case rather rain low quality bug reports down upon the poor browser developers, we spend our limited time trying to make CryptPad better. In the case of ServiceWorker in Firefox, our conclusion is that in effect the technology remains experimental and shouldn’t be relied upon.

SharedWorker to the rescue

Fortunately there is another technology called SharedWorker which is essentially identical to a plain vanilla WebWorker but can communicate with all tabs which are navigated to the site. Unfortunately this technology is only supported in Chrome and Firefox, but the support for this technology, we found, really works!

However, Firefox still has the issue with of multiple SharedWorkers being created for multiple tabs, but since this issue is fixable we were able to go ahead with it anyway. For browsers which have no SharedWorker, they would fallback to plain old WebWorker. Though this seems like it would be a problem, it is in fact quite fine because actions done in the drive by another WebWorker are the same as actions done in the drive by another device, they need to be encrypted and sent to the server in order to persist anyway.

Faster CryptPad

Coming in the release on Tuesday June 26, 2018, we will have a new SharedWorker based CryptPad instance, that means when you use Chrome or Firefox, the first time you open CryptPad, it will load your drive, but then every tab you open after that will communicate with the SharedWorker managing the drive and therefore pads will load nearly twice as fast.

Signing CryptPad

CryptPad was designed with a view that privacy should be default and cryptography should be invisible. In order to do this, we made use of the web-app model so people could just go to cryptpad.fr and immediately begin using the app, no installation necessary. However, this model has a known flaw, the server can decide what client-side code it will send to any given user, allowing a compromized server to serve code with a back-door vulnerability.

Recently, I did an experiment to make CryptPad more secure against these types of attacks by signing the code. CryptPad is a unique webapp, even without considering the encryption aspect. There is no build system, the code we write is exactly the same as what your web browser runs. All of the CryptPad html, javascript and resources are static files which are served by a plain old web server. The data persistance is managed by an API server which the web browser communicates with using an HTML5 WebSocket. Finally, in order to add a layer of security against possible Cross Site Scripting attacks, CryptPad makes use of a cross domain iframe, protecting your encryption keys from the majority of the CryptPad code in the same way that your online bank is protected from that sketchy porn site open in another tab.

Since CryptPad has no build system, there are many small javascript files which must be loaded. To do this, CryptPad uses RequireJS. While many small files are generally considered to be bad for website performance, RequireJS uses the HTML5 async attribute to tell the browser not to block loading of other things while waiting for the scripts to load. Secondly, RequireJS also allows version numbers to be added to the script URLs which allows us to cache almost everything in the browser. Finally, we use the HTTP/2 protocol to serve resources because it allows multiple requests to be sent at the same time, while HTTP/2 is incompatible with WebSocket, this is ok because the web-app is served from a different server from the API server.

Chain of Trust

Just one corrupted script is enough to render the security of an entire web-app useless, so in CryptPad we needed the signing to cover all javascript files. Fortunately there is a new HTML5 technology called Subresource Integrity which allows putting the hash of a script in a script tag attribute and makes the browser verify the script before executing it.

Insecure, some.website can serve you anything:

1
2
3
<script
src="https://some.website/path/to/script.js"
></script>

Secure, only one possible script can be sent by some.website or else the web browser will throw an error and refuse to run the script:

1
2
3
4
<script
src="https://some.website/path/to/script.js"
integrity="sha256-G1KwaJYUEDsA1SD/6Wt4z0laskKzIwgqgs5cYH0CW/o="
></script>

So rather than signing every script, I only needed to make a list of hashes of every script, and sign that. What I needed was a way to generate a manifest, and so I developed a small program which could hash all of the javascript files in CryptPad and generate a manifest file. The content of the manifest looks something like this:

1
2
3
4
5
6
7
8
9
"files": {
"assert": {
"frame": {
"frame.js": "BrN2JNnK4QJCztw3PyRRPAsEwSq5lczTBrRkzdLAFow=",
"respond.js": "yO0KFMHiCdE1fXFWPVaFB+Mmh37OCl/UNPpXrYtWF7A="
},
"main.js": "ABf3uhmYVHWaHX6vhK8K2jAUY8XqRjjMJ2FqXVGLZE0=",
"translations": {
"main.js": "50Ami2eghyXcGKGYTaDK1vUeEuAEG7kcpvUoCKbUaUU="

It contains a JSON tree which mirrors the files that are part of the CryptPad codebase and the hashes of the files for the Subresource Integrity check. Once the manifest.js file was created, then I needed a javascript file which would load and verify it. Since the manifest is different every time a new release is made, the verification of the manifest needed to be via signing. The manifest hash was signed along with a version number and those were placed in a file called version.txt and version.txt is loaded using a file called sboot.js. The hash of sboot.js was included directly into the html files which are cached, so sboot.js can never be changed at all.

Loading process

index.html

First, the browser loads the html file, the html file contains a single script tag loading sboot.js

1
2
3
4
5
6
<script
async
data-bootload="/customize/template.js"
src="/common/sboot.js?ver=8IaxCUqjpzoP7AEPEk%2B%2BVQ%2BBk83mRdXx4dK%2BXvSNPcI%3D"
integrity="sha256-8IaxCUqjpzoP7AEPEk++VQ+Bk83mRdXx4dK+XvSNPcI="
></script>

There is a custom attribute called data-bootload which indicates which javascript file should be loaded for that html file.

sboot.js

When sboot.js gets loaded, it downloads and then verifies version.txt which is a signed message containing the CryptPad version number and the hash of manifest.js. The content that is signed looks something like this:

1
[85,"h+tOXVmYBWMmiVDylXvnRq28LWRVs6xy+goBwNEELZk="]

The version number (85) is not the CryptPad version but rather an auto-incrementing number which is stored in the browser localStorage and prevents the server from downgrading the version of CryptPad. After the signature/version check completes successfully, sboot.js loads manifest.js like the following:

1
2
3
4
5
<script
async
src="/customize/manifest.js?ver=h%2BtOXVmYBWMmiVDylXvnRq28LWRVs6xy%2BgoBwNEELZk%3D"
integrity="sha256-h+tOXVmYBWMmiVDylXvnRq28LWRVs6xy+goBwNEELZk="
></script>

You will notice that the hash is used also in the URL of manifest.js, this allows the server to signal that the files are immutable and can be cached by the browser forever which makes CryptPad load faster next time.

After manifest.js loads, sboot.js finds the hash of require.js in the manifest and then manually loads require.js in the same way. Once require.js is loaded, sboot.js configures require to use the hashes from the manifest for every file it loads, then it uses require to load boot2.js.

boot2.js

This file is not needed for security, but unlike sboot.js, it can easily be changed from release to release and it contains any code which should be run before the main CryptPad code. Things such as additional requirejs configuration and shims for missing browser APIs are placed here. After boot2.js is complete, it reads the data-bootload attribute from the html file and invokes require to load that.

Further development

While this system provides excellent security, it is still not perfect. If the root html file is compromized then it can alter the chain of trust, or scrap it completely. With a very long cache header, the browser will store the html file essentially forever, but if the user triggers a hard reload with the F5 key, then the cache will be flushed.

The root html file can be signed using pgp and then verified using the signed pages chrome extension. But signed pages is not able to prevent the loading of the website even if the signature is invalid and it only takes 1 second for the keys in localStorage to be leaked.

If the root html file was generated by the server each load, it could contain a secret key which is used to encrypt the keys in the localStorage, thus rendering them unusable if the html file is re-loaded, and meaning that the user must re-enter their password and would then be able to see that the signature on the html file is invalid, however unless signed pages can ignore the key inside of the html file when verifying the signature, it would have to be re-signed every time, pushing the pgp key onto the server, which we are worried about being compromized.

There are also a number of configuration files in the CryptPad project which are in fact javascript files and would thus be signed by the release manager, preventing anyone hosting CryptPad from changing them so it may be a long time before this project is merged into CryptPad mainline, however it is available and you can experiment with it by checking out the code-integrity branch of the CryptPad project.