Move more development information to the dev docs to minimise repitition and getting out of sync

apidocs/dev/README.md

@@ -2,8 +2,112 @@
 
 This site provides information that is useful for developing the OpenFlexure Microscope.
 
-### Viewing and Modifying Locally
+## Viewing and Modifying These Docs Locally
 
 If you want to view these documents in your machine (very useful when editing them). You will need to serve the `apidocs` directory. Navigate to the `apidocs` directory and run `python -m http.server`.
 
-All developer guidance documentation is written in markdown and then displayed with Docsify. More information on Docsify can be found at https://docsify.js.org.
+All developer guidance documentation is written in markdown and then displayed with Docsify. More information on Docsify can be found at https://docsify.js.org.
+
+
+## Installation and set up
+
+To set up a development environment you will need Python 3.11 and the static files for the web application.
+
+### Installation and set up on a Raspberry Pi
+
+The OpenFlexure Raspberry Pi OS image comes with Python 3.11 already installed. We do not recommend trying to develop on the standard Raspberry Pi OS image.
+
+
+* To activate the microscope virtual environment run `ofm activate`
+* Navigate to the Openflexure repository with `cd /var/openflexure/application/openflexure-microscope-server`
+
+The Pi should already come with the static files for the web application.
+
+### Installation and set up on other platforms
+
+You should install Python 3.11. If you have a different version of python we recommend using [UV](https://docs.astral.sh/uv/) to set up a Python 3.11 virtual environment.
+
+You can check your Python version by running `py --version` or `python --version` in any terminal. Your Python version should be >= 3.11.0.
+
+* Clone the source code: `git clone https://gitlab.com/openflexure/openflexure-microscope-server.git`
+* Enter the directory `cd openflexure-microscope-server`
+* Create the virtual environment, for example with `python -m venv .venv` (or with `uv venv --python 3.11` if using uv)
+* Activate your virtual environment with `source .venv/bin/activate` (on Linux) or `.venv\Scripts\activate.bat` (on Windows CMD) or `.venv\Scripts\activate` (on Windows Powershell)
+
+_Note: If you activate the venv in a CMD terminal in VSCode on Windows, the venv will not visually show as activated until you open a new powershell window._
+
+* Install the application `pip install -e .[dev]` (or `uv pip install -e .[dev]` if you are using UV)
+* Run `python pull_webapp.py` to get the latest static files for the web application.
+
+Full video walkthroughs of the software installation process are available below:
+
+* [Windows Walkthrough](https://www.youtube.com/watch?v=LhnOLlIk-b8)
+* [Linux Walkthrough](https://youtu.be/81jkMKOBeHQ)
+
+> If you are using Windows and you cannot activate your virtual environment from powershell, you probably need to change the execution policy. The official Microsoft docs have the right command in example #1: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.security/set-executionpolicy?view=powershell-7.5#example-1-set-an-execution-policy.
+
+## Running the server
+
+There are two types of server available to run:
+
+* A live OpenFlexure Microscope Server, that runs on a Raspberry Pi within your OpenFlexure Microscope. This server processes real images from the microscope's camera.
+
+   | ![The View Tab in the UI for the OFM Server](/assets/ofm_microscope_ui_view.jpeg "The View Tab in the UI for the OFM Server") | ![The Slide Scan Tab in the UI for the OFM Server](/assets/ofm_microscope_ui_scan.jpeg "The Slide Scan Tab in the UI for the OFM Server")|
+   |:---:|:---:|
+
+* A Simulation Server, which replicates the software functionality of the live server, but instead runs on your computer. Instead of receiving a live feed from a camera, the server 'scans' simulated data from a simulated camera.
+
+   | ![The View Tab in the UI for the Simulated Server](/assets/ofm_sim_ui_view.jpg "The View Tab in the UI for the Simulated Server") | ![The Slide Scan Tab in the UI for the Simulated Server](/assets/ofm_sim_ui_scan.jpg "The Slide Scan Tab in the UI for the Simulated Server")|
+   |:---:|:---:|
+
+
+### Running the server on a Raspberry Pi
+
+You can manage the server with the `ofm` command, using `ofm start`, `ofm stop`, and `ofm restart` to do the respective actions. Often, stopping the server and running it manually will make errors easier to spot. To do this, run:
+```
+ofm stop
+ofm activate
+cd /var/openflexure
+sudo -u openflexure-ws openflexure-microscope-server --fallback -c microscope_configuration.json --host 0.0.0.0 --port 5000
+```
+
+### Running the simulation server on other platforms
+
+To start the simulation server:
+
+* Ensure all steps of the installation for your platform of choice are complete.
+* Activate the virtual environment that was set up during installation
+* Run the following command:
+
+`openflexure-microscope-server --fallback -c ./ofm_config_simulation.json`
+
+* Open the address displayed in the terminal in any browser.
+* To close the simulation server, return to the terminal and press `ctrl + c`.
+
+> Note: If you are using Windows, we advise running this command in PowerShell. If it doesn't work this, might be a permission issue on managed machines, in which case try executing the command in a CMD terminal.
+
+> Note: If you receive an error saying that port 5000 is already in use, you can specify a different port to run the simulation on using `--port`:
+>
+>    `openflexure-microscope-server --fallback --port 8081 -c ./ofm_config_simulation.json`
+
+A [full video walkthrough of running the simulation server](https://youtu.be/Z8verCqfj9s) is available on our YouTube Channel.
+
+You can find more information on using the simulation server in the [Simulation Guide](simulation_guide).
+
+
+## General development overview
+
+Please first read our **[project contributions guide](contributing)**
+
+* We recommend all development is done on a branch.
+* If you are editing the web app, please see our see the README in the [webapp](webapp) directory.
+* If you want to try a branch with webapp changes (that are already pushed to GitLab), but don't want to compile the javascript, then run `python pull_webapp.py -b <branch_name>` to get the artifacts from the GitLab CI.
+
+**For development on a Raspberry Pi** we recommend VSCode Remote session from another computer.  This allows you to use your usual developer environment to write code, but everything runs on the Raspberry Pi with real hardware.  Note that the repository is cloned by default over `https`, so you will probably need to change the "remote" URL to your fork of the repository, or to SSH, before you're able to push changes.
+
+### Adding functionality
+
+The microscope comprises a number of `Thing` instances, which provide actions and properties to implement hardware control and software features. These may be imported from any Python module, using `ofm_config.json`. See the LabThings-FastAPI documentation for how to create a `Thing`. 
+
+Currently, the camera and stage classes are provided by external libraries, `labthings-picamera2` and `labthings-sangaboard`. Replacing these is the easiest way to use alternative hardware: interfaces are defined in `openflexure_microscope_server.things.camera` and `openflexure_microscope_server.things.stage` respectively.
+

apidocs/dev/_sidebar.md

@@ -5,5 +5,9 @@
 <h2>Developer Guidance<h2>
 
 - [Home](README)
-- [Software Architecture](architecture.md)
-- [PiCamera Tuning Files](picamera-tuning-files.md)
+- [Contributing](contributing)
+- [Python Development](python)
+- [Webapp Development](webapp)
+- [Running the Simulator](simulation_guide)
+- [Software Architecture](architecture)
+- [PiCamera Tuning Files](picamera-tuning-files)

apidocs/dev/contributing.md

@@ -0,0 +1,36 @@
+# Contributing to the OpenFlexure Microscope Server
+
+All contributions to the OpenFlexure project should follow our [project contributions guidelines](https://openflexure.org/contribute).
+
+Any contributions to any of our projects should follow [our code of conduct](https://openflexure.org/conduct).
+
+If you are interested in contributing to the OpenFlexure project, and do not know where to start, we recommend you visit [our forum](https://openflexure.discourse.group/).
+
+**The rest of this page details contribution guidelines that are specific to this repository.**
+
+This page generally focusses on the back-end of the server, not the front-end webapp. For more details on the webapp development see the [webapp's README](webapp).
+
+## Installation
+
+For installation, see the [home page](README).
+
+### Core packages and concepts
+
+The OpenFlexure Microscope server is built in the [LabThings-FastAPI framework](https://labthings-fastapi.readthedocs.io/en/latest/). This provides a convenient way to expose laboratory hardware over HTTP via a [FastAPI server](https://fastapi.tiangolo.com/).
+
+The core elements of the server code are `Things`. A `Thing` creates a [W3C Web of Things](https://www.w3.org/TR/wot-architecture) compliant HTTP interface for hardware interactions and other server functionality.
+
+The project also makes use of the following libraries:
+
+* [Pydantic](https://docs.pydantic.dev/latest/) - For serialising/deserialising data being saved to disk or sent over HTTP.
+* [OpenFlexure Stitching](https://gitlab.com/openflexure/openflexure-stitching) - Our own library for creating composite microscope images from a scan.
+* [OpenCV](https://opencv.org/) and [Picamera2](https://github.com/raspberrypi/picamera2/) - For controlling cameras, and simulating cameras for tests.
+* [Pillow](https://pillow.readthedocs.io) - For image manipulation and saving.
+* [NumPy](https://numpy.org/) and [SciPy](https://scipy.org/) - For mathematical calculations.
+
+*The list above is not exhaustive!*
+
+## Coding guidelines
+
+We have separate coding guidelines for [the Python server](python) and for the [Vue webapp](webapp).
+

apidocs/dev/python.md

@@ -0,0 +1,92 @@
+# Python Server Development
+
+## Overview
+
+
+Ideally, Python code should be:
+
+* Broken up into small, clear, testable functions.
+* Documented with docstrings (For conventions see below).
+* Type hinted
+* Covered by unit tests (using [pytest](https://docs.pytest.org))
+
+The server code is going through significant flux during the transition from v2 to v3, so currently our test coverage is being rebuilt and type checking does not yet pass.
+
+## Python environment, build, and dependencies
+
+As of `v3` we specify dependencies in `pyproject.toml`. These are not currently frozen due to difficulties matching versions on different platforms. On Raspberry Pi, it's best to specify `--only-binary=:all:` to ensure libraries like `numpy` and `scipy` are not compiled from source (which takes many hours, and/or fails). Pinning dependencies with `requirements.txt` and/or `requirements.in` may happen in the future.
+
+
+
+## Python: Formatting, linting, and tests
+
+[Ruff](https://docs.astral.sh/ruff/) is used both for linting and formatting the codebase. Our custom linter rules are in our `pyproject.toml`.
+
+> All of the commands below assume that you are running in the OFM virtual environment, i.e. you have run `ofm activate` on an OpenFlexure SD card, or `source .venv/bin/activate` on Linux, or `.venv/Scripts/activate`on Windows Powershell, or `.venv\Scripts\activate.bat` on Windows CMD.
+
+**Before committing** you should lint and auto-format your code:
+
+* To lint run `ruff check`
+* To auto-format the Python code run `ruff format`
+
+**Before merging** please auto-format your code and also run the quality checks (linting, static analysis, and unit tests)
+
+* All commands above
+* `pytest`
+
+
+We use several code analysis and formatting libraries in this project. Our CI will check each of these automatically, so ensuring they pass locally will save you time. Currently `ruff` is used for linting/formatting and `pytest` for unit tests. `mypy` will be enabled once the codebase is ready.
+
+
+To check your code before each commit we recommend putting the following script in tour `.git/hooks` directory with the filename `pre-commit`.
+
+
+```python
+#!/bin/bash
+
+set -e
+
+function angrymessage()
+{
+    RED='\033[1;31m'
+    NC='\033[0m' # No Color
+    printf "\n${RED}Not committing as ruff failed${NC}\n\n"
+}
+
+trap angrymessage ERR
+
+ruff format --check
+ruff check
+```
+
+
+## Docstrings conventions.
+
+We write docstrings in [reStructuredText](https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html), and process them with [pydoctor](https://pydoctor.readthedocs.io/).
+
+For function arguments, return values, and exceptions we use the following style recognised by reStructuredText:
+
+```python
+def a_function(arg1: str, arg2: int) -> tuple[float, int]:
+    """A one sentence summary, in imperative mood, ending with punctuation.
+
+    More detail after a blank line. This could be a whole paragraph.
+
+    When referring to ``code`` in reStructuredText used double backticks
+
+    :param arg1: This is the first argument, we can say everything we need to about it.
+        If the text wraps onto a new line, this line is indented.
+    :param arg2: This is the second argument.
+
+    :returns: Say what it returns without anything within the colons. If returning
+        multiple values, a bulleted list is a good idea - just make sure:
+
+        * It is indented to 1 level in
+        * There is a blank line before the bullets
+        * The bullet points are ``*`` not ``-``
+    """
+```
+
++The linter rules we use for docstrings are defined by [pydocstyle](https://docs.astral.sh/ruff/rules/#pydocstyle-d). We do not use rules D203, D213, D400 for reasons explained in our `pyproject.toml` file.
+
+The documentation generated by pydoctor from docstrings can be viewed in any merge request (if the CI pipeline has passed) using the "View App" button.

apidocs/dev/simulation_guide.md

@@ -0,0 +1,88 @@
+# Guide: Running the OpenFlexure Microscope Simulation Server
+
+This guide walks you through running the simulation server for the OpenFlexure Microscope, along with useful tips for using the simulation features.
+
+The simulation server GUI is a copy of the GUI you would find on the hardware with a simulated sample you can explore. Most functionality included in the hardware operates on the simulation server. This includes:
+
+* Camera stage mapping and 'click to move'
+* Scanning and stitching*
+* Autofocus
+* Image capture
+* Stage navigation
+
+**Please note: the final stitch at the end of a scan currently doesn't work on Windows.*
+
+---
+
+## Starting the Simulation Server
+
+1. **Activate your virtual environment** (the same one created during installation, as described in the [Python documentation](python)).
+2. **Run the simulation server** using the following command:
+
+   ```bash
+   openflexure-microscope-server --fallback -c ./ofm_config_simulation.json
+   ```
+
+   > Note: If you are using Windows, we advise running this command in PowerShell. If it doesn't work this, might be a permission issue on managed machines, in which case try executing the command in a CMD terminal.
+
+This can fail with an error that the chosen port is already in use. In that case, append `--port 8081`
+
+---
+
+## Video Walkthrough
+
+For a step-by-step demonstration, watch the [full video walkthrough on running the simulation server](https://youtu.be/Z8verCqfj9s).
+
+---
+
+## Running a simulated scan
+
+To run a simulated scan, you must first complete the calibration wizard.
+
+1. Go to the  **Slide Scan** tab
+2. Click the **Start Smart Scan** Button.
+3. This will scan until the whole sample has been imaged. You can cancel the scan at any time, with the **Cancel** button.
+
+---
+
+## Simulation Settings and Tips
+
+---
+
+### Auto-calibration
+
+* Your simulated microscope should prompt you to complete the auto-calibration wizard upon your first use of the simulated server.
+* You can press the **Escape** key at any point to exit the wizard, but this will leave your simulated microscope in an uncalibrated state.
+* If your microscope is not calibrated, the wizard will launch each time you load the UI, prompting you to complete remaining calibrations.
+* You can always launch the full calibration wizard by going to: **Settings → Launch Calibration Wizard**.
+
+The calibration steps are:
+
+1. The Camera Calibration will set the background image needed for background detect.
+  * You can rerun this calibration by going to: **Settings → Camera → Full Auto-Calibrate**.
+2. The Camera-Stage Mapping calibration is used to find the relationship between stage and image coordinates.
+  * If the stage moves in the wrong direction when double clicking on the camera feed in the **View** or **Navigate** tabs, you may need to re-run Camera-Stage Mapping.
+  * To re-run Camera-Stage Mapping, go to: **Settings → Camera to Stage Mapping → Auto-Calibrate Using Camera**.
+
+
+---
+
+### Background Detect
+
+* You can **load a sample** to simulate imaging conditions, using the **Load Sample** options in the **Settings tab**.
+* To remove a sample and reset the view, use the **Remove Sample** option in the **Settings tab**.
+* You can load and remove simulated samples under: **Settings → Camera → Load/Remove Sample**.
+* The Full Auto-Calibrate option in camera settings can be used to **Remove the sample → Run background detection → Re-load the sample**.
+* You can adjust the sample coverage needed for the image not to be detected as background in the **Background Detect tab** under: **Configure → Sample Coverage**. The default value is 25%.
+
+---
+
+### Scans
+
+* If you perform a scan with **0 images** (i.e. cancelling the scan before any images are collected), it will **disappear** from the scan list as soon as a new scan starts.
+* Scans are automatically given **Scan IDs** in numerical order if the user does not provide one.
+
+---
+
+If you need further help, reach out to the community on the [OpenFlexure Forum](https://openflexure.discourse.group/).
+

apidocs/dev/webapp.md

@@ -0,0 +1,35 @@
+# WebApp Development
+
+## Key info
+
+* Vue.js web application providing a graphical interface for the Openflexure Microscope.
+* Once built, will be served by the API server from the host root on port 5000
+* JS client is coupled to the API, and is built and distributed with the server.
+
+
+## Installing
+
+_Note: If you are using a Windows OS, the following commands must be executed in a Powershell terminal._
+
+* Install Node.js v18 (and npm)
+* Navigate into the webapp directory with `cd webapp`
+* Install dependencies with `npm install`
+* Build the static web app with `npm run build`
+
+## Live Server
+
+When developing it is useful to use the development server so Vue changes happen instantly without the need to rebuild. To start the development server run:
+
+    npm run serve
+
+The development server is accessed on a different port from the microscope API. The port to access the development server is printed on the command line when you run the above command.
+
+When the development webapp starts it cannot locate the microscope API as this is served by the microscope on port 5000. Instead it will present you with a page giving you the option to "override API origin". If you are running a development server on your computer, you should enter `http://localhost:5000/`.
+
+# Javascript: Formatting and linting
+
+To enforce code style we and quality we use ESLint and Prettier. Both can be run together with the same command.
+
+- To check the for errors and warnings run `npm run lint`
+- To automatically fix errors an warnings run `npm run lint:fix`
+