Update documentation on dependencies/dev tools

Pipfile

@@ -1,13 +0,0 @@
-[[source]]
-url = "https://www.piwheels.org/simple"
-verify_ssl = true
-name = "piwheels"
-
-[requires]
-python_version = '3.7'
-
-[packages]
-openflexure-microscope-server = {editable = true, path = "."}
-
-[dev-packages]
-openflexure-microscope-server = {editable = true, path = ".", extras = ["dev"]}

README.md

@@ -85,7 +85,7 @@ To set up a development version of the software (most likely using emulated came
 ### Run the server manually on a Raspberry Pi
 ```
 cd /var/openflexure
-sudo -u openflexure-ws PATH="/var/openflexure/application/openflexure-microscope-server/.venv/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin" /var/openflexure/application/openflexure-microscope-server/.venv/bin/uvicorn --reload --reload-dir=./application --host 0.0.0.0 --port 5000 openflexure_microscope_server.server:app
+sudo -u openflexure-ws PATH="/var/openflexure/application/openflexure-microscope-server/.venv/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin" /var/openflexure/application/openflexure-microscope-server/.venv/bin/openflexure-microscope-serve -c /var/openflexure/settings/ofm_config.json --host 0.0.0.0 --port 5000
 ```
 
 
@@ -107,70 +107,36 @@ sudo -u openflexure-ws PATH="/var/openflexure/application/openflexure-microscope
 ## Formatting, linting, and tests
 All of the commands below assume that you are running in the OFM virtual environment, i.e. you have run `ofm activate` on an OpenFlexure SD card, or `source .venv/bin/activate` on Linux, or `.venv/Scripts/activate`on Windows.
 
-**Before committing** you should auto-format your code:
+**Before committing** you should lint and auto-format your code:
 
-* To auto-format the Python code run `poe format`
+* To lint run `ruff check`
+* To auto-format the Python code run `ruff format`
 * To auto-format the Javascript code, run
   * `cd webapp`
   * `npm run lint`
 
-**Before submitting a merge request/merging** please auto-format your code and also run the quality checks (linting, static analysis, and unit tests)
+**Before merging** please auto-format your code and also run the quality checks (linting, static analysis, and unit tests)
 
-* To auto-format and type-check the Python code run `poe check`
+* All commands above
-* To auto-format the Javascript code, run
+* `mypy src` (currently fails)
-  * `cd webapp`
+* `pytest`
-  * `npm run lint`
 
 
 ### Details
 
-We use several code analysis and formatting libraries in this project. **Please run all of these before submitting a merge request.**
+We use several code analysis and formatting libraries in this project. Our CI will check each of these automatically, so ensuring they pass locally will save you time. Currently `ruff` is used for linting/formatting and `pytest` for unit tests. `mypy` will be enabled once the codebase is ready.
-
-Our CI will check each of these automatically, so ensuring they pass locally will save you time.
-
-* **Black** - Code formatting with minimal configuration.
-  * While sometimes it's not perfect, its fine 90% of the time and prevents arguments about formatting.
-  * Automatically formats your code
-  * This will rewrite your files in-place, so if you want to be able to revert, make a backup first!
-  * `poe black`
-* **Pylint** - Static code analysis
-  * Analyses your code, failing if issues are detected.
-  * We've disabled some less severe warnings, so _if anything fails your merge request will be blocked_
-  * `poe pylint`
-* **Mypy** - Type checking
-  * Analyses your type hints and annotations to flag up potential bugs
-  * Where possible, use type hints in your code. Even if dependencies don't support it, it'll help identify issues.
-  * `poe mypy`
-* **Pytest** - Unit testing
-  * While unit testing is of limited use due to our dependence on real hardware, some simple isolated functions can (and should) be unit tested.
-  * `poe test`
-
-Though not in the CI, our `format` script also runs isort:
-
-* **Isort** - Import sorting
-  * Automatically organises your imports to stop things getting out of hand
-  * `poe isort`
 
 ## Python environment, build, and dependencies
 
-As of `2.10.0b0` we have switched to using `pipenv` for managing dependencies, and a standard `setuptools` based build system.  See "local installation" above for instructions on how to install the project.  Earlier versions of the project used `poetry` but we moved away because of [difficulties](https://gitlab.com/openflexure/openflexure-microscope-server/-/merge_requests/124) getting it to work both on the Raspberry Pi and in our CI pipeline.  The new arrangement for configuration files is:
+As of `v3` we specify dependencies in `pyproject.toml`. These are not currently frozen due to difficulties matching versions on different platforms. On Raspberry Pi, it's best to specify `--only-binary=:all:` to ensure libraries like `numpy` and `scipy` are not compiled from source (which takes many hours, and/or fails). Pinning dependencies with `requirements.txt` and/or `requirements.in` may happen in the future.
-
-* Dependencies, and dev-dependencies, are specified in `setup.py` in the usual way (using `install_requires` and `extra_requires[dev]`).
-* Package metadata is specified in `setup.py` in the usual way.
-* `pyproject.toml` is retained, but *no longer includes package metadata or dependencies* and does not have a `[tool.poetry]` table.  It does define the build system as per PEP517, which is `setuptools`, and it also contains settings for `black`, `poe` and `isort`.
-* `Pipfile` is very minimal, and only declares dependencies on the current module, i.e. the dependencies declared in `setup.py`.  It also specifies the Python version.  This allows us to single-source dependency information from `setup.py` but use the dependency resolution/locking functionality of `pipenv`
-* `Pipfile.lock` locks the dependency versions in the same way as `poetry.lock` used to, i.e. it specifies exact versions of everything, derived from the looser specifications in `setup.py`.
-* Dependency resolution is a pain, in part because compiling packages like scipy and numpy from source on a Pi is slow and unreliable. To avoid this, we usually lock the dependencies on a Raspberry Pi, using:
-  `PIP_ONLY_BINARY="numpy, scipy, matplotlib" pipenv lock --dev`
-  This forces the use of wheels only, and thus restricts us to things that built OK on Piwheels.  The resulting Pipfile.lock should then work nicely on other Raspberry Pis.
-* **The Pipfile we use is now Raspberry Pi-specific** so if you attempt to use it on other platforms it will probably fail. This is because we've removed `pypi` to avoid hash conflicts. The CI now runs on a Raspberry Pi image, so that the tests use a similar environment to our deployment.  Currently, changing the `[source]` entry back to PyPi will allow you to re-lock the dependencies on other platforms. In the future we may need to maintain two Pipfiles and two Pipfile.lock files.
 
 ## Creating releases
 
 * Update the application's internal version number
-    * Edit `setup.py` to update the version number
+  * Edit `pyproject.toml` to update the version number
-    * Git commit and git push
+* Update the changelog
-* Create a new version tag on GitLab (e.g. `v2.6.11`)
+* Git commit and git push
+* Create a new version tag on GitLab (e.g. `v2.6.11`) that matches the `pyproject.toml` version number.
     * Make sure you prefix a lower case 'v', otherwise it won't be recognised as a release!
     * This tagging will trigger a CI pipeline that builds the JS client, tarballs up the server, and deploys it
         * Note: This also updates the build server's nginx redirect map file
@@ -180,10 +146,8 @@ As of `2.10.0b0` we have switched to using `pipenv` for managing dependencies, a
 * `npm install -g conventional-changelog-cli`
 * `npx conventional-changelog -r 1 --config ./changelog.config.js -i CHANGELOG.md -s`
 
-## Microscope extensions
+## Adding functionality
 
-The Microscope module, and Flask app, both support plugins for extending lower-level functionality not well suited to web API calls. The current documentation can be found [here](https://openflexure-microscope-software.readthedocs.io/en/latest/plugins.html).
+The microscope comprises a number of `Thing` instances, which provide actions and properties to implement hardware control and software features. These may be imported from any Python module, using `ofm_config.json`. See the LabThings-FastAPI documentation for how to create a `Thing`. 
-If you want to add functions to the microscope software, this is probably the best mechanism to use if it works for you.
 
-
+Currently, the camera and stage classes are provided by external libraries, `labthings-picamera2` and `labthings-sangaboard`. Replacing these is the easiest way to use alternative hardware: interfaces are defined in `openflexure_microscope_server.things.camera` and `openflexure_microscope_server.things.stage` respectively.
-```

pyproject.toml

@@ -29,6 +29,9 @@ dependencies = [
 [project.optional-dependencies]
 dev = [
   "labthings-fastapi[dev]",
+  "ruff",
+  "mypy",
+  "pytest",
 ]
 pi = [
   "labthings-picamera2 == 0.0.1-dev2",

webapp/README.md

@@ -11,7 +11,7 @@
 
 ## Creating releases
 
-* JS client is coupled to the API, and so are no longer separately built and deployed.
+* JS client is coupled to the API, and is built and distributed with the server.
 * See [openflexure-microscope-server/README.md](https://gitlab.com/openflexure/openflexure-microscope-server/-/blob/master/README.md) for details on creating new releases
 
 ## Installing