Update documentation on dependencies/dev tools
This commit is contained in:
Richard Bowman
parent
9fd8fc37f1
commit
77794c94eb
5 changed files with 21 additions and 2454 deletions
13
Pipfile
13
Pipfile
|
|
@ -1,13 +0,0 @@
|
|||
[[source]]
|
||||
url = "https://www.piwheels.org/simple"
|
||||
verify_ssl = true
|
||||
name = "piwheels"
|
||||
|
||||
[requires]
|
||||
python_version = '3.7'
|
||||
|
||||
[packages]
|
||||
openflexure-microscope-server = {editable = true, path = "."}
|
||||
|
||||
[dev-packages]
|
||||
openflexure-microscope-server = {editable = true, path = ".", extras = ["dev"]}
|
||||
2387
Pipfile.lock
generated
2387
Pipfile.lock
generated
File diff suppressed because it is too large
Load diff
70
README.md
70
README.md
|
|
@ -85,7 +85,7 @@ To set up a development version of the software (most likely using emulated came
|
|||
### Run the server manually on a Raspberry Pi
|
||||
```
|
||||
cd /var/openflexure
|
||||
sudo -u openflexure-ws PATH="/var/openflexure/application/openflexure-microscope-server/.venv/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin" /var/openflexure/application/openflexure-microscope-server/.venv/bin/uvicorn --reload --reload-dir=./application --host 0.0.0.0 --port 5000 openflexure_microscope_server.server:app
|
||||
sudo -u openflexure-ws PATH="/var/openflexure/application/openflexure-microscope-server/.venv/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin" /var/openflexure/application/openflexure-microscope-server/.venv/bin/openflexure-microscope-serve -c /var/openflexure/settings/ofm_config.json --host 0.0.0.0 --port 5000
|
||||
```
|
||||
|
||||
|
||||
|
|
@ -107,70 +107,36 @@ sudo -u openflexure-ws PATH="/var/openflexure/application/openflexure-microscope
|
|||
## Formatting, linting, and tests
|
||||
All of the commands below assume that you are running in the OFM virtual environment, i.e. you have run `ofm activate` on an OpenFlexure SD card, or `source .venv/bin/activate` on Linux, or `.venv/Scripts/activate`on Windows.
|
||||
|
||||
**Before committing** you should auto-format your code:
|
||||
**Before committing** you should lint and auto-format your code:
|
||||
|
||||
* To auto-format the Python code run `poe format`
|
||||
* To lint run `ruff check`
|
||||
* To auto-format the Python code run `ruff format`
|
||||
* To auto-format the Javascript code, run
|
||||
* `cd webapp`
|
||||
* `npm run lint`
|
||||
|
||||
**Before submitting a merge request/merging** please auto-format your code and also run the quality checks (linting, static analysis, and unit tests)
|
||||
**Before merging** please auto-format your code and also run the quality checks (linting, static analysis, and unit tests)
|
||||
|
||||
* To auto-format and type-check the Python code run `poe check`
|
||||
* To auto-format the Javascript code, run
|
||||
* `cd webapp`
|
||||
* `npm run lint`
|
||||
* All commands above
|
||||
* `mypy src` (currently fails)
|
||||
* `pytest`
|
||||
|
||||
|
||||
### Details
|
||||
|
||||
We use several code analysis and formatting libraries in this project. **Please run all of these before submitting a merge request.**
|
||||
|
||||
Our CI will check each of these automatically, so ensuring they pass locally will save you time.
|
||||
|
||||
* **Black** - Code formatting with minimal configuration.
|
||||
* While sometimes it's not perfect, its fine 90% of the time and prevents arguments about formatting.
|
||||
* Automatically formats your code
|
||||
* This will rewrite your files in-place, so if you want to be able to revert, make a backup first!
|
||||
* `poe black`
|
||||
* **Pylint** - Static code analysis
|
||||
* Analyses your code, failing if issues are detected.
|
||||
* We've disabled some less severe warnings, so _if anything fails your merge request will be blocked_
|
||||
* `poe pylint`
|
||||
* **Mypy** - Type checking
|
||||
* Analyses your type hints and annotations to flag up potential bugs
|
||||
* Where possible, use type hints in your code. Even if dependencies don't support it, it'll help identify issues.
|
||||
* `poe mypy`
|
||||
* **Pytest** - Unit testing
|
||||
* While unit testing is of limited use due to our dependence on real hardware, some simple isolated functions can (and should) be unit tested.
|
||||
* `poe test`
|
||||
|
||||
Though not in the CI, our `format` script also runs isort:
|
||||
|
||||
* **Isort** - Import sorting
|
||||
* Automatically organises your imports to stop things getting out of hand
|
||||
* `poe isort`
|
||||
We use several code analysis and formatting libraries in this project. Our CI will check each of these automatically, so ensuring they pass locally will save you time. Currently `ruff` is used for linting/formatting and `pytest` for unit tests. `mypy` will be enabled once the codebase is ready.
|
||||
|
||||
## Python environment, build, and dependencies
|
||||
|
||||
As of `2.10.0b0` we have switched to using `pipenv` for managing dependencies, and a standard `setuptools` based build system. See "local installation" above for instructions on how to install the project. Earlier versions of the project used `poetry` but we moved away because of [difficulties](https://gitlab.com/openflexure/openflexure-microscope-server/-/merge_requests/124) getting it to work both on the Raspberry Pi and in our CI pipeline. The new arrangement for configuration files is:
|
||||
|
||||
* Dependencies, and dev-dependencies, are specified in `setup.py` in the usual way (using `install_requires` and `extra_requires[dev]`).
|
||||
* Package metadata is specified in `setup.py` in the usual way.
|
||||
* `pyproject.toml` is retained, but *no longer includes package metadata or dependencies* and does not have a `[tool.poetry]` table. It does define the build system as per PEP517, which is `setuptools`, and it also contains settings for `black`, `poe` and `isort`.
|
||||
* `Pipfile` is very minimal, and only declares dependencies on the current module, i.e. the dependencies declared in `setup.py`. It also specifies the Python version. This allows us to single-source dependency information from `setup.py` but use the dependency resolution/locking functionality of `pipenv`
|
||||
* `Pipfile.lock` locks the dependency versions in the same way as `poetry.lock` used to, i.e. it specifies exact versions of everything, derived from the looser specifications in `setup.py`.
|
||||
* Dependency resolution is a pain, in part because compiling packages like scipy and numpy from source on a Pi is slow and unreliable. To avoid this, we usually lock the dependencies on a Raspberry Pi, using:
|
||||
`PIP_ONLY_BINARY="numpy, scipy, matplotlib" pipenv lock --dev`
|
||||
This forces the use of wheels only, and thus restricts us to things that built OK on Piwheels. The resulting Pipfile.lock should then work nicely on other Raspberry Pis.
|
||||
* **The Pipfile we use is now Raspberry Pi-specific** so if you attempt to use it on other platforms it will probably fail. This is because we've removed `pypi` to avoid hash conflicts. The CI now runs on a Raspberry Pi image, so that the tests use a similar environment to our deployment. Currently, changing the `[source]` entry back to PyPi will allow you to re-lock the dependencies on other platforms. In the future we may need to maintain two Pipfiles and two Pipfile.lock files.
|
||||
As of `v3` we specify dependencies in `pyproject.toml`. These are not currently frozen due to difficulties matching versions on different platforms. On Raspberry Pi, it's best to specify `--only-binary=:all:` to ensure libraries like `numpy` and `scipy` are not compiled from source (which takes many hours, and/or fails). Pinning dependencies with `requirements.txt` and/or `requirements.in` may happen in the future.
|
||||
|
||||
## Creating releases
|
||||
|
||||
* Update the application's internal version number
|
||||
* Edit `setup.py` to update the version number
|
||||
* Git commit and git push
|
||||
* Create a new version tag on GitLab (e.g. `v2.6.11`)
|
||||
* Edit `pyproject.toml` to update the version number
|
||||
* Update the changelog
|
||||
* Git commit and git push
|
||||
* Create a new version tag on GitLab (e.g. `v2.6.11`) that matches the `pyproject.toml` version number.
|
||||
* Make sure you prefix a lower case 'v', otherwise it won't be recognised as a release!
|
||||
* This tagging will trigger a CI pipeline that builds the JS client, tarballs up the server, and deploys it
|
||||
* Note: This also updates the build server's nginx redirect map file
|
||||
|
|
@ -180,10 +146,8 @@ As of `2.10.0b0` we have switched to using `pipenv` for managing dependencies, a
|
|||
* `npm install -g conventional-changelog-cli`
|
||||
* `npx conventional-changelog -r 1 --config ./changelog.config.js -i CHANGELOG.md -s`
|
||||
|
||||
## Microscope extensions
|
||||
## Adding functionality
|
||||
|
||||
The Microscope module, and Flask app, both support plugins for extending lower-level functionality not well suited to web API calls. The current documentation can be found [here](https://openflexure-microscope-software.readthedocs.io/en/latest/plugins.html).
|
||||
If you want to add functions to the microscope software, this is probably the best mechanism to use if it works for you.
|
||||
The microscope comprises a number of `Thing` instances, which provide actions and properties to implement hardware control and software features. These may be imported from any Python module, using `ofm_config.json`. See the LabThings-FastAPI documentation for how to create a `Thing`.
|
||||
|
||||
|
||||
```
|
||||
Currently, the camera and stage classes are provided by external libraries, `labthings-picamera2` and `labthings-sangaboard`. Replacing these is the easiest way to use alternative hardware: interfaces are defined in `openflexure_microscope_server.things.camera` and `openflexure_microscope_server.things.stage` respectively.
|
||||
|
|
|
|||
|
|
@ -29,6 +29,9 @@ dependencies = [
|
|||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"labthings-fastapi[dev]",
|
||||
"ruff",
|
||||
"mypy",
|
||||
"pytest",
|
||||
]
|
||||
pi = [
|
||||
"labthings-picamera2 == 0.0.1-dev2",
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@
|
|||
|
||||
## Creating releases
|
||||
|
||||
* JS client is coupled to the API, and so are no longer separately built and deployed.
|
||||
* JS client is coupled to the API, and is built and distributed with the server.
|
||||
* See [openflexure-microscope-server/README.md](https://gitlab.com/openflexure/openflexure-microscope-server/-/blob/master/README.md) for details on creating new releases
|
||||
|
||||
## Installing
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue