54 lines
7.6 KiB
Plaintext
54 lines
7.6 KiB
Plaintext
== Introduction
|
|
// TO-DO: Grammar-check. I'm totally fried right now and can't seem to complete even a single proper
|
|
*{software-family}* - short for _**e**LabFTW to Ne**X**us **Pars**er_ - is (hopefully) a family of specialized parsing software applications, mainly developed in Python, whose primary job is to automatically transform experimental metadata and data - originally stored as JSON objects inside an electronic lab notebook - into standardized, self-descriptive **NeXus files**.
|
|
|
|
The software is designed to fetch "scattered" data (often distributed across multiple linked entries) from our eLNfootnote:[Acronym for "_electronic Lab Notebook_".] of choice - link:{elabftw-site}[**eLabFTW**^] - where the data is originally stored as JSON objects. It then parses the included metadata to resolve the full dataset which is then used to create a dictionary following a pre-established schema (dependent on the analysis or fabrication method, e.g., PLD, XRD, or RHEED), and finally uses said dictionary to produce an **HDF5/NeXus file** which complies with the **FAIR Principles** and the guidelines given within the context of the Italian PNRRfootnote:pnrr[PNRR stands for _National Recovery and Resilience Plan_.] **NFFA-DI**.
|
|
|
|
Specifically, *{software-name}* is designed for *Pulsed Laser Deposition / PLD* fabrications.
|
|
|
|
=== NFFA-DI and FAIR Principles
|
|
PNRR (_Piano Nazionale di Ripresa e Resilienza_) is Italy's national recovery plan from the aftermaths of COVID-19. +
|
|
*NFFA-DI* (_Nano Foundries and Fine Analysis - Digital Infrastructure_) is a project within this plan aimed at creating a distributed digital infrastructure for nanoscience and nanotechnology. In practice, NFFA-DI provides a unified cyber-platform for researchers to access advanced instrumentation, simulation tools, and data management services across multiple Italian research centers.
|
|
|
|
Like most modern scientific projects NFFA-DI is _FAIR by design_, meaning it strives for total compliance to *FAIR Principles*. FAIR is the acronym of the four main characteristics all compliant projects should share:
|
|
|
|
> * Findable: «Metadata and data should be easy to find for both humans and computers.»
|
|
> * Accessible: «Once the user finds the required data, she/he/they need to know how they can be accessed, possibly including authentication and authorisation.»
|
|
> * Interoperable: «The data usually need to be integrated with other data. In addition, the data need to interoperate with applications or workflows for analysis, storage, and processing.»
|
|
> * Reusable: «Metadata and data should be well-described so that they can be replicated and/or combined in different settings.»
|
|
>
|
|
> Source: link:{go-fair-site}[GO FAIR^]
|
|
|
|
|
|
{software-name} contributes to NFFA-DI goals by enabling automated data harmonization: converting local PLD experiment records into a common, shareable format (NeXus) with a mutually agreed upon schema, thereby making the data interoperable across the entire NFFA-DI ecosystem.
|
|
|
|
TIP: More info on NFFA-DI at link:{nffa-di-site}[nffa-di.it^].
|
|
|
|
=== eLabFTW
|
|
*eLabFTW* is an open-source, web-based electronic laboratory notebook and resource manager. It acts as a central digital hub for one or more laboratories, organizing information (as database entries) into two main constructs:
|
|
|
|
* **Experiments**: They are the core feature of eLabFTW, can contain structured data (via custom JSON fields), unstructured text, timestamps, tags, links to files (attachments), and relations to database items.
|
|
* **Resources** or **Items**: This is a separate, structured inventory for items like raw materials (targets, substrates), instruments (UHV machines) or samples. Each entry is built from customizable templates with defined metadata (e.g. for a substrate batch we have name, manufacturer, geometry, available pieces left...).
|
|
|
|
Although separated into different database constructs, experiments and items all have their own unique, incremental internal ID, which we'll simply call *elabid* to distinguish it from other identifiers, with no academic utility but extremely important when dealing with eLabFTW from a developer's perspective.
|
|
|
|
// method-specific
|
|
In a software like eLabFTW where data can (and will) be spread out through multiple entries, a particularly useful feature is **linking**: the software allows you to link experiments or items with each other, using elabid's as identifiers. For a PLD deposition, you can link the experiment describing a single layer to the target used, the substrate, the PLD instrument and the sample produced itself (all of which are eLabFTW items). This creates a complete provenance graph which can be (not-so) easily resolved starting from the sample's metadata and a chain of HTTP requests.
|
|
|
|
In this optic, {software-name} interacts with eLabFTW via its REST API (Application Programming Interface). It reads a starting sample's ID (the entry point), fetches the relevant JSON metadata, chains requests using the elabid's of the sample's linked resources and experiments, rebuilds the entire dataset and if available downloads attached instrument files (e.g., RHEED intensities, images) to package all of it into the final NeXus file.
|
|
|
|
=== The output: HDF5 and NeXus files
|
|
The output of {software-family} is an **HDF5 (Hierarchical Data Format ver. 5) file**, which is a powerful file format designed to store and organize large volumes of numerical data. It acts like a virtual file system inside a single file, using a hierarchical group/dataset structures in the same way a file system uses folders and files - with both elements having their own metadata; this way the file is self-describing, containing all relevant information like a small database. HDF5 also supports efficient slicing, compression and parallel I/O. The file extension of such format is `.h5`.
|
|
|
|
On the other hand, *NeXus* is a common data standard [.underline]#built on top of HDF5#. It defines fixed conventions for naming groups, datasets and attributes, specifically for neutron, X-ray, and now materials science experiments. NeXus provides "application definitions" (like _NXpld_fabrication_ for PLD) that specify exactly which fields must/may appear. NeXus is also heavily promoted by _FAIRmat_, a German-based consortium, part of the NFDI, whose main mission is providing scientists «with a FAIR data infrastructure and the skills and tool they need to make the most of it»footnote:[As stated on their link:{fairmat-site}[website^].]. The file extension of such format is `.nxs`, but generally file viewers treat the two formats similarly.
|
|
|
|
Last but not least, NeXus is also the format of choice for data sharing in the NFFA-DI guidelines. Which brings us to the reason why {software-family} exists.
|
|
|
|
[#reading-nxs]
|
|
==== Reading HDF5/NeXus files
|
|
While writing an HDF5/NeXus file usually requires dedicated software and/or a good knowledge of programming and familiarity with specific libraries (like h5py), there are multiple ways to read these files even without such knowledge.
|
|
|
|
One of such ways would be using the online NeXus file viewer of the NCNR (_NIST Center for Neutron Research_), available on their link:{ncnr-viewer}[website^]. The "_Browse..._" button at the bottom allows for uploading both h5 and nxs files, although drag and drop also works.
|
|
|
|
Another similar but in my opinion more elegant online file viewer is the one hosted by the HDF5 Group: link:{hdf5-viewer}[MyHDF5^]. Other than the more modern appearance this viewer doesn't upload files to any remote server, with every operation happening locally in your browser; the drag and drop works better meaning you won't accidentally reload the page if you miss the dropping area, and the viewer also allows for opening multiple concurrent files, and downloading h5 files from URL.
|