Added automatic upload of tags to Zenodo.

Co-authored-by: Kaspar Emanuel <kaspar@monostable.co.uk>

.gitlab-ci.yml

@@ -230,3 +230,26 @@ deploy:
   only:
     - tags
     - web
+
+zenodo:
+  stage: zenodo
+  image: ubuntu:20.04
+  before_script:
+    - apt-get update -qq
+    - apt-get -y -qq install git python3-pip
+    - pip3 install -r scripts/zenodo/requirements.txt
+  variables:
+    ZENODO_USE_SANDBOX: "true"
+  script:
+    - git archive "${CI_COMMIT_REF_NAME}" --format=zip --output="${CI_PROJECT_NAME}-${CI_COMMIT_REF_NAME}-source.zip"
+    - python3 scripts/zenodo/upload_to_zenodo.py "${CI_PROJECT_NAME}-${CI_COMMIT_REF_NAME}-source.zip" dist/*
+  artifacts:
+    # this is only a small html link, let's just keep it forever
+    # gitlab doesn't understand "expire_in: never" yet though
+    expire_in: 100000 years
+    name: zenodo-${CI_PROJECT_NAME}-${CI_COMMIT_REF_NAME}-link.html
+    paths:
+      - zenodo-link.html
+  only:
+    - tags
+    - web

scripts/zenodo/README.md

@@ -0,0 +1,45 @@
+# Zenodo archiving
+This folder contains some scripts put together by Kaspar Bumke, and re-used by Richard, to upload the contents of the repository, together with any CI build artefacts, to Zenodo.  The result is a link, taking you to a pre-populated upload on Zenodo that you can manually correct and upload.  In order to customise it for a new project, there are a few steps you need, outlined below.
+
+## Setting up archival
+* Copy this folder to `scripts/zenodo` in your repository.
+* Customise `metadata.yaml` (in this folder) with the metadata you want to be attached to your uploads.  Zenodo provide [documentation](https://developers.zenodo.org/#deposit-metadata) on the contents of this file, but hopefully the one here will be a useful start.
+* Add a CI job to `.gitlab-ci.yml` at the top level of your repository.  The one in this repository should serve as an example.  This requires you to add a couple of places:
+  - Ensure you have a "stage" called "deploy" that runs after you've built anything you will want to archive.  This is the last stage by default, but if you have a custom list of stages, it is usually found at the top of the file.  For example:
+    ```yaml
+    stages:
+    - analysis
+    - testing
+    - build
+    - package
+    - deploy
+    ```
+  - Copy and paste the block below, to build and upload the source and built artefacts.
+    ```yaml
+    zenodo:
+    stage: zenodo
+    image: ubuntu:20.04
+    before_script:
+        - apt-get update -qq
+        - apt-get -y -qq install git python3-pip
+        - pip3 install -r scripts/zenodo/requirements.txt
+    variables:
+        ZENODO_USE_SANDBOX: "true"
+    script:
+        - git archive "${CI_COMMIT_REF_NAME}" --format=zip --output="${CI_PROJECT_NAME}-${CI_COMMIT_REF_NAME}-source.zip"
+        - python3 scripts/zenodo/upload_to_zenodo.py "${CI_PROJECT_NAME}-${CI_COMMIT_REF_NAME}-source.zip" dist/*
+    artifacts:
+        # this is only a small html link, let's just keep it forever
+        # gitlab doesn't understand "expire_in: never" yet though
+        expire_in: 100000 years
+        name: zenodo-${CI_PROJECT_NAME}-${CI_COMMIT_REF_NAME}-link.html
+        paths:
+        - zenodo-link.html
+    only:
+        - tags
+        - web
+    ```
+  - Customise the lines that create archives or specify files for upload.  As a minimum, you will need to customise the list of files to upload.  The block above will upload everything in the `dist/` folder, which is populated with archives produced in previous CI jobs.  If you don't have any other CI jobs, you can just delete the `dist/*` from the end of the relevant line, and the script will upload an archive of the source code only.
+  - **Note: LFS files are not included in the source archive.** If you use LFS, you may want to make the source archive manually in CI, so that LFS files are downloaded and included.  If you don't know what LFS is, you can probably ignore this.  If you look at your upload, and there are lots of files that should be large, and are actually less than 200 bytes, you probably do have LFS enabled.
+* Generate a Zenodo access token.  There are instructions in the [Zenodo documentation](https://developers.zenodo.org/#authentication), but in brief you must first log in to Zenodo and [generate an access token](https://zenodo.org/account/settings/applications/tokens/new/).  If you want to try it out using the "sandbox" before uploading to Zenodo proper, you can do the same thing with the [sandbox site](https://sandbox.zenodo.org/account/settings/applications/tokens/new/).  You need to tick `deposit:write` on the token's permissions.
+* Paste your Zenodo access token into a [GitLab CI variable](https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project) called `ZENODO_API_KEY_REAL`, and paste the sandbox access token into another variable called `ZENODO_API_KEY_SANDBOX`.

scripts/zenodo/metadata.yaml

@@ -0,0 +1,20 @@
+# see https://developers.zenodo.org/#deposit-metadata
+title: OpenFlexure Microscope Software
+upload_type: other
+description: >
+  Software to run an OpenFlexure Microscope.  This consists of a Python 
+  server, and a web application client.
+creators:
+  - name: 'Collins, Joel T.'
+    affiliation: 'University of Bath'
+    orcid: '0000-0002-0167-4969'
+  - name: 'Stirling, Julian'
+    affiliation: 'University of Bath'
+    orcid: '0000-0002-8270-9237'
+  - name: 'Bumke, Kaspar'
+    affiliation: 'University of Bath'
+  - name: 'Bowman, Richard'
+    affiliation: 'University of Bath'
+grants:
+  - id: "10.13039/501100000690::EP/R011443/1"
+  - id: "10.13039/501100000690::EP/R013969/1"

scripts/zenodo/requirements.txt

@@ -0,0 +1,2 @@
+PyYAML==5.3.1
+requests==2.23.0

scripts/zenodo/upload_to_zenodo.py

@@ -0,0 +1,59 @@
+import os
+import requests
+from argparse import ArgumentParser, Namespace
+from zenodo import Zenodo
+import yaml
+
+# you have to explicitely set ZENODO_USE_SANDBOX=false to not use the
+# sandbox, any other value or unset variable means this script will use
+# the sandbox zenodo site
+if "ZENODO_USE_SANDBOX" in os.environ:
+    USE_SANDBOX = not os.environ["ZENODO_USE_SANDBOX"] == "false"
+else:
+    USE_SANDBOX = True
+
+
+if USE_SANDBOX:
+    API_KEY = os.environ["ZENODO_API_KEY_SANDBOX"]
+else:
+    API_KEY = os.environ["ZENODO_API_KEY_REAL"]
+
+
+def parse_arguments() -> Namespace:
+    p = ArgumentParser(description="Upload data to Zenodo")
+    p.add_argument("paths", help="Directories and files to upload to Zenodo", nargs="*")
+    return p.parse_args()
+
+
+def script_directory(path):
+    """resolves path to directory of the current script"""
+    return os.path.join(os.path.dirname(os.path.realpath(__file__)), path)
+
+
+def get_meta():
+    with open(script_directory("metadata.yaml")) as f:
+        metadata = f.read()
+
+    return yaml.safe_load(metadata)
+
+
+def main():
+    args = parse_arguments()
+
+    metadata = get_meta()
+
+    zenodo = Zenodo(API_KEY, USE_SANDBOX)
+    deposit = zenodo.create_new_deposit()
+
+    zenodo.set_metadata(deposit["id"], metadata)
+
+    for path in args.paths:
+        zenodo.upload_file(deposit["id"], path)
+
+    link = deposit["links"]["latest_draft_html"]
+    with open("zenodo-link.html", "w") as f:
+        f.write(f'<a href="{link}">{link}</a>')
+
+
+if __name__ == "__main__":
+    main()

scripts/zenodo/zenodo.py

@@ -0,0 +1,103 @@
+"""
+Copyright (c) 2020 Bath Open Instrumentation Group
+Adapted from: https://gitlab.com/schlauch/zenodo-api-test/
+Copyright (c) 2018 German Aerospace Center (DLR). All rights reserved.
+SPDX-License-Identifier: MIT-DLR
+
+"""
+import json
+import os
+
+import requests
+
+
+class Zenodo:
+    def __init__(self, api_token, use_sandbox=True):
+        self._api_token = api_token
+        self._use_sandbox = use_sandbox
+        if use_sandbox:
+            self.zenodo_url = "https://sandbox.zenodo.org/api/deposit/depositions"
+        else:
+            self.zenodo_url = "https://zenodo.org/api/deposit/depositions"
+
+    def create_new_deposit(self):
+        """ Creates a new (unpublished) Zenodo deposit and return its deposition ID. """
+
+        headers = {"Content-Type": "application/json"}
+        r = requests.post(
+            self.zenodo_url,
+            params={"access_token": self._api_token},
+            json={},
+            headers=headers,
+        )
+        print(r.status_code)
+        print(r.json())
+        return r.json()
+
+    def set_metadata(self, deposition_id, metadata):
+        """ Sets the given metadata for the specified deposit. """
+
+        headers = {"Content-Type": "application/json"}
+        r = requests.put(
+            self.zenodo_url + "/{}".format(deposition_id),
+            params={"access_token": self._api_token},
+            data=json.dumps({"metadata": metadata}),
+            headers=headers,
+        )
+        print(r.status_code)
+        print(r.json())
+
+    def upload_file(self, deposition_id, file_path):
+        """ Uploads a new file for the given deposit. """
+
+        file_name = os.path.basename(file_path)
+        data = {"filename": file_name}
+        files = {"file": open(file_path, "rb")}
+        r = requests.post(
+            self.zenodo_url + "/{}/files".format(deposition_id),
+            params={"access_token": self._api_token},
+            data=data,
+            files=files,
+        )
+        print(r.status_code)
+        print(r.json())
+
+    def publish_deposit(self, deposition_id):
+        """ Publishes the given deposit. BEWARE: It is now visible to all!!! """
+
+        r = requests.post(
+            self.zenodo_url + "/{}/actions/publish".format(deposition_id),
+            params={"access_token": self._api_token},
+        )
+        print(r.status_code)
+        print(r.json())
+
+    def create_new_version(self, deposition_id):
+        """ Creates a new version of an already published deposit. """
+
+        r = requests.post(
+            self.zenodo_url + "/{}/actions/newversion".format(deposition_id),
+            params={"access_token": self._api_token},
+        )
+        print(r.status_code)
+        print(r.json())
+        return os.path.basename(r.json()["links"]["latest_draft"])
+
+    def remove_all_files(self, deposition_id):
+        """ Removes all uploaded files of a unpublished deposit. """
+
+        r = requests.get(
+            self.zenodo_url + "/{}/files".format(deposition_id),
+            params={"access_token": self._api_token},
+        )
+        print(r.status_code)
+        print(r.json())
+        for file_entry in r.json():
+            print("Remove file entry: " + file_entry["id"])
+            print(
+                requests.delete(
+                    self.zenodo_url
+                    + "/{}/files/{}".format(deposition_id, file_entry["id"]),
+                    params={"access_token": self._api_token},
+                )
+            )