Software Metadata Extraction and Curation Software (SMECS)

A web application to extract and curate research software metadata following the CodeMeta (version 3.0) software metadata standard.

SMECS facilitates the creation and curation of research software metadata by extracting metadata from GitHub and GitLab repositories or importing existing metadata. It provides a user-friendly graphical interface for reviewing, editing, and completing the metadata without unnecessarily re-entering information already available elsewhere. The curated metadata are exported as CodeMeta-compliant JSON, supporting integration with other tools and the discoverability and reuse of research software.

📄 For more details, see our Paper.

Authors: Stephan Ferenz @sferenz, Aida Jafarbigloo @AidaJafarbigloo

Phases in SMECS

The workflow of SMECS consists of four phases: Start, Extraction, Curation, and Export.

Depending on the selected input method, the Extraction phase may be skipped.

1. Start Phase

In the Start phase, users select how metadata should be provided to SMECS. Three input methods are available:

a) Extract metadata from a repository

This is the default workflow. Users provide:

• A repository URL from GitHub or GitLab
• A personal access token for the corresponding platform

SMECS uses the repository information to automatically extract available software metadata. The extracted metadata are then passed to the Curation phase.

For some repositories, SMECS can perform the extraction without a user-provided token by using an internally configured token. However:

• For GitLab instances, a user-provided token is required.
• Providing a personal access token may allow SMECS to access and extract additional repository metadata.

b) Import a local metadata file

Users who already have software metadata can upload an existing CodeMeta JSON file. In this case, the extraction phase is skipped and the imported metadata are loaded directly into the Curation phase.

c) Paste metadata content

Users can also paste existing CodeMeta JSON content directly into SMECS. As with file import, the Extraction phase is skipped and the provided metadata are loaded directly into the Curation phase for review and editing.

2. Extraction Phase

The Extraction phase is used when users select metadata extraction from a GitHub or GitLab repository in the Start phase. If users import a local metadata file or paste existing metadata content, this phase is skipped and the provided metadata are passed directly to the Curation phase.

For repository-based input, SMECS uses the harvesting functionality of HERMES to retrieve metadata from multiple sources. For details on the metadata fields, see: Metadata Terms in SMECS.

SMECS uses four HERMES harvesters:

• GitHub
• GitLab
• CFF (Citation File Format)
• CodeMeta

GitHub and GitLab metadata are harvested via the HERMES GitHub/GitLab plugin.

All harvested metadata are mapped to CodeMeta using existing crosswalks from CodeMeta and HERMES, together with a custom crosswalk for GitLab.

The harvested metadata are then processed and merged through the HERMES processing step, producing a unified metadata set that is passed to the Curation phase.

The HERMES-based approach provides an interoperable and modular extraction architecture and facilitates the integration of additional harvesting sources in the future.

3. Curation Phase

The Curation phase allows users to review, edit, and refine the metadata obtained through either repository-based extraction or metadata import.

The metadata are displayed in a form-based interface organized into four main tabs:

1. General Information
2. Provenance
3. Related Persons
4. Technical Aspects

Key visualization and curation features include:

• Metadata Visualization & User-Friendly Interface: Metadata are displayed in a structured, easy-to-read format. The interface is intuitive, responsive, and allows smooth navigation through metadata fields.
• Missing Metadata Identification: SMECS flags fields where metadata is absent.
• Required Metadata Properties: Certain fields are marked as mandatory to ensure completeness of the final output.
• Editable Fields: Users can directly edit or correct metadata within the interface.
• Tagging Feature: Some fields allow multiple values for better metadata organization.
• Suggestion Lists: For selected fields, SMECS provides suggestions to reduce manual input and ensure consistency.
• Form-to-JSON Synchronization: Updates in the form are mirrored in the JSON view (one-directional) so users can track changes instantly.

4. Export Phase

In the Export phase, the curated metadata can be downloaded as a CodeMeta 3.0–compliant JSON file.

The exported file can, for example, be:

• Included in the software repository
• Used as input for other tools or services
• Uploaded to a software registry

Installation and Usage

Install from GitHub

• Cloning the repository

git clone https://github.com/NFDI4Energy/SMECS.git

• Navigate to the Project Directory

cd SMECS

• Creating virtual environment

  • Ensure that Python 3.10 or higher is installed on your system.

    • Windows: Check the version with py --version.
    • Unix/MacOS: Check the version with python3 --version.

  • Create the virtual environment.

    • Windows:

    py -m venv my-env

    • Unix/MacOS:

    python3 -m venv my-env

    for more details visit Creation of virtual environments

  • Activate virtual environment.

    • Windows:

    my-env\Scripts\activate

    • Unix/MacOS:

    source my-env/bin/activate

    (Note that activating the virtual environment change the shell's prompt and show what virtual environment is being used.)

• Managing Packages with pip

  • Ensure you can run pip from command prompt.

    • Windows:

    py -m pip --version

    • Unix/MacOS:

    python3 -m pip --version

  • Update pip Packages if required.

    • Windows:

    py -m pip install --upgrade pip

    • Unix/MacOS:

    python3 -m pip install --upgrade pip

  • Install the required packages listed in Requirements.txt.

    • Windows:

    py -m pip install -r requirements.txt

    • Unix/MacOS:

    python3 -m pip install -r requirements.txt

  for more details visit Installing Packages

• Running the project

  • Set up the database.

    • Apply the database migrations. Django stores the database structure as migration files in the repository. This command applies those migrations to create your local database with all the tables the application needs, such as captcha, sessions, and admin. Run it once during setup, and again whenever new migrations are added later.

    • Windows:

    py manage.py migrate

    • Unix/MacOS:

    python3 manage.py migrate

  • Run the project.

    • Windows:

    py manage.py runserver

    • Unix/MacOS:

    python3 manage.py runserver

• To see the output on the browser follow the link shown in the terminal. (e.g. http://127.0.0.1:8000/)

Install through Docker

To get started with SMECS using Docker, follow the steps below:

• Prerequisites: Make sure Docker is installed on your local machine.
• Cloning the Repository

git clone https://github.com/NFDI4Energy/SMECS.git

• Navigate to the Project Directory

cd SMECS

• Building the Docker Images

docker-compose build

• Starting the Services

docker-compose up

• Accessing the Application: Navigate to http://localhost:8000 in your web browser.
• Stopping the Services

docker-compose down

Setting Up GitLab/GitHub Personal Token

Depending on the repository and hosting platform, SMECS may require a personal access token to retrieve repository metadata. Providing a personal access token may also enable access to additional metadata that is not available without authentication.

• Generate a GitLab Token:

  • Visit Create a personal access token for more information on how to generate a new token.

• Generate a GitHub Token:

  • Visit Managing your personal access tokens for more information on how to generate a new token.

Tip for developers

If the page does not refresh correctly, clear the browser cache. You can force Chrome to pull in new data and ignore the saved ("cached") data by using the keyboard shortcut Cmd+Shift+R on Mac, and Ctrl+F5 or Ctrl+Shift+R on Windows.

Collaboration

We believe in the power of collaboration and welcome contributions from the community to enhance the SMECS workflow. Whether you have found a bug, have a feature idea, or want to share feedback, your contribution matters. Feel free to submit a pull request, open up an issue, or reach out with any questions or concerns.

To see upcoming features in SMECS, please refer to our open issues.

To stay updated on upcoming changes to the HERMES GitHub and GitLab Plugin, visit the project’s issues page. And if you have questions, suggestions, feedback, or need to report a bug, please open a new issue there.

License and Citation

The code is licensed under the GNU Affero General Public License v3.0 or later (AGPL-3.0-or-later).

See LICENSE.txt for further information.

Acknowledgements

The authors would like to thank the German Federal Government, the German State Governments, and the Joint Science Conference (GWK) for their funding and support as part of the NFDI4Energy consortium within the German National Research Data Infrastructure (NFDI, www.nfdi.de). This work has been partially funded by the German Research Foundation (DFG) with grants No. 561044496 corresponding to the ConnOSS project and No. 501865131 corresponding to NFDI4Energy.

The authors also thank the meta_tool for providing the foundational framework upon which this project is built.