We need a free and convenient electronic lab notebook

Posted on by Guillaume 9 mins read

Here are a few thoughts about the way I use lab notebooks, why I think there is currently no satisfying software to use as a convenient electronic lab notebook (ELN), and why the scientific community really needs one.

How do I use lab notebooks?

In my PhD lab, the official lab notebook was a paper notebook. We could record results however we wanted, but in the end we had to keep the formal lab notebook up to date no matter what. I personnaly used a small notebook for daily quick record-keeping in the lab, and I copied its content in the formal notebook at least weakly (daily when I had enough time). This was the most convenient procedure for me for several reasons:

  • the formal notebook was too large to conveniently carry it around with me in the lab;
  • I needed to be able to read records in this formal notebook even months after an experiment, and my handwriting in spontaneous record conditions is simply too bad for this purpose: I really had to organize my notes and make them readable after the initial act of (messily) recording them;
  • my desk and the lab were on two different floors, so using an ELN was not an option (I already travelled way too often between those two floors because I regularly forgot a pen or calculator or whatever).

Now, in my post-doc lab, things are very different. First, my desk is 5 meters away from my bench! This is very convenient: I can now use an ELN without having to walk the entire building to record something. Second, the lab encourages the use of an ELN and accepts periodic printed dumps for archiving, whereas in my former lab I could technically do the same but I felt this was not really encouraged. Here is how I currently keep records:

  • I use Microsoft OneNote because it’s provided by the university, and it’s actually very convenient and suitable as an ELN despite several important flaws (more on that later in this post);
  • because I don’t like my ELN being captive in a proprietary format, I also keep my notes in a local instance of Laverna in parallel: this only involves regular copy/paste from OneNote to Laverna (really not a big hassle), but Laverna’s user interface is really poor compared to OneNote so I won’t use it as my main ELN. It is only a backup to have my notes locally and in an open format.
  • Finally, I also try as much as possible to keep a paper lab notebook (PLN) up to date, because paper has a huge advantage over computer files for long-term archiving: we can still read papyrus from ancient Egypt after thousands of years. Try reading today a file created 30 years ago with the first version of MS Word and stored on a floppy disk: good luck with that. Maintaining this PLN is very time consumming though, so maybe at some point I will stop doing that and peridodically print dumps of my ELN instead.

What an ideal ELN should look like?

Here, I will emphasize the biggest advantages of ELN versus PLN, and what an ideal ELN should offer.

The biggest advantages of ELN are:

  • they are searchable;
  • it’s easy to make cross-references with links between pages, sections or paragraphs;
  • they are readable (compared to some people’s handwriting…);
  • they are shareable (you can also share a PLN, but only with one person at a time, or making photocopies but it’s slow and virtually nothing compared to publishing an ELN online);
  • they are printable, so they even have the best advantage of PLN.

The biggest advantages of PLN are:

  • they are the most reliable and time-tested option for long-term archiving;
  • they are the most reliable option for legal purposes (proving who did something first, patenting issues, etc.), provided records are kept correctly! This point is important because with a PLN the user has to do things correctly (date, sign and make a witness date and sign), whereas an ELN could automate these things to some extent (at least digital signing and timestamping could be easily automated; to make a witness date and sign, one still needs to find a witness willing to check the lab notebook records);
  • writing on paper allows one to quickly draw complicated schemes to clarify ideas, and drawing is more difficult in an ELN (this can be solved with tablet computers and digital pens, but that goes beyond the standard tools available in the average lab).

An ideal ELN should have both of those two sets of advantages, plus:

  1. be free software (I mean free as in freedom) and cross-platform;
  2. allow saving local files in an open format (that usually goes well with the first point);
  3. be installable locally by the user, or as a server for an entire lab;
  4. allow one to easily append all sort of data (images, tables, etc.) to an entry: no experimental science discipline only produces text;
  5. enforce proper timestamping without user intervention. This is important to prove when an experiment was performed for the first time;
  6. enforce digital signature without user intervention (except maybe first-time setup needed to create a signing key: the user would have to indicate their identity). This is important to prove authorship and data integrity;
  7. enforce read-only status for all records older than the current day, without user intervention. An ideal lab notebook should not allow its user to delete or modify previous records, but only allow to append new records.

Points 1, 2 and 3 are important to ensure that notes will never be captive in a proprietary format or in a proprietary cloud (a “cloud” is simply someone else’s computer; I can’t find who first coined this expression, but it describes the problem very well). Point 4 is important because if the user interface is not convenient, users will keep poor records of their research, with predictable bad consequences. Points 5, 6 and 7 are critical for legal purposes, and are easy tasks to automate so they really shouldn’t be left to the user (everyone forgets to do things from time to time).

Why most current ELN solutions are pretty disappointing?

Now I will explain why, from my limited experience, there is currently no satisfactory solution for keeping an ELN.

Most, if not all, specialized commercial products are proprietary software storing data in a proprietary format or cloud. That’s not a good option, and I won’t even comment on those products. One can get an idea simply looking for “electronic lab notebook” in their favorite search engine.

Microsoft OneNote has a nice user interface, but many flaws:

  • it does not comply with points 1, 2, 5, 6 and 7 outlined above, above all not being able to have a local copy of files is really disturbing (I have to trust that nothing bad will happen “in the cloud”);
  • in its current version, it has an annoying behavior regarding language settings: the spellchecker defaults to the language detected from the keyboard layout, which is a real pain (I use a French AZERTY keyboard so OneNote keeps spellckecking in French by default even if my languages settings are set to English and my records are written in English). This cannot be changed for an entire notebook, one has to set the spellchecker language paragraph by paragraph! This is driving me crazy but I have no better alternative ELN right now (read on to discover why).

Evernote looks very similar to OneNote. I didn’t try it, but from what I read about it I guess it would have more or less the same flaws as OneNote (except maybe this insane spellchecker thing).

Laverna is free software, but its user interface is really poor, and overall it does not comply with points 4, 5, 6 and 7. This is only a backup solution for me right now, because I can’t stand my ELN being locked somewhere in Microsoft Office365 cloud.

This software called eLabFTW seems promising: it complies with many of the points outlined above (according to its website). The main problem is that it is primarily intended to be installed on a server, and therefore doesn’t fully comply with point 3. An individual user can install it locally but it seems to be a big hassle for the average user, and it actually discouraged me to even try this software…

Thinking about it further, an ELN can actually be set up using several specialized tools together. A git repository can provide timestamping and revision history, and even publishing online if coupled with a hosting site (like Framagit and other GitLab instances, or Bitbucket, or GitHub). A good text editor can provide the main user interface to enter records, I use GNU Emacs for this. Digital signing can be achieved with GnuPG and can even be automated for each git commit. Keeping records in simple flat text files, one can export to more elaborate printable formats with pandoc, optionally automating the process with make (useful to ensure the final exported file is actually up to date without having to go through the entire file list by hand). All these tools are free software: this workflow therefore complies with points 1 and 2 above. Point 3 is easily achieved by setting up a central git repository, should an entire lab decide to adopt this solution. Point 5 is partially achieved, because git commit automatically records a timestamp, but the user still has to run the command (it’s not straightforward to mimick the auto-save feature of OneNote with git). Point 6 needs some setup, but is fully automated and transparent for the user once the initial setup is done! Digital signing also provides timestamping, but it also depends on the user running the commit command. Point 7 is not easy to achieve and automate with simple tools (like filesystem permissions), but with a workflow based on a git repository the revision history allows one to easily detect modifications, therefore the need for read-only entries is not critical. The main problem with this workflow is that it makes it unnecessarily cumbersome to add records, and especially to add non-text data to a record: to enter a new record, one has to choose a file name and location; to add an image, one has to save the image somewhere and insert a link to the image in the text file constituting the associated record. Compare this to OneNote where one simply writes text and literally drops everything else in: way easier.

We need a free software electronic lab notebook

I use a light version (I don’t use digital signature) of the workflow described in the previous paragraph to keep a collection of protocols, because they are only text and not updated as often as a lab notebook, which makes them convenient to manage this way. But I wouldn’t use this workflow for an ELN where I need to attach many types of heterogeneous data (images, tables, schemes, etc.), and need to add records on a daily basis: this requires way too much effort to do properly with this stratedy, and I believe a good dedicated software would be way more efficient for this task. The ideal ELN would offer a user interface similar to OneNote, without its many flaws, and with all the specific features required for a lab notebook (timestamping, signing, etc.).

This is a call to the free software community: we scientists need a convenient and free ELN! :-)