Move hardware-specific-tests to tests dir

tests/hardware_specific_tests/README.md

@@ -0,0 +1,132 @@
+# Hardware specific tests
+
+This directory contains unit tests that require specific hardware to run. Current they implement tests for code that interacts with the Raspberry Pi Camera.
+
+## Hardware specific tests for the Pi Camera.
+
+The OpenFlexure Microscope Server runs automated test in the cloud (via the GitLab CI) to test each version of the code as changes are proposed and merged. As these tests run in the cloud they do not have access to actual microscope hardware. As such, these tests either test specific blocks of code that don't need hardware access or make use of simulated hardware via the simulation camera and dummy stage.
+
+To test the code that interacts with the hardware itself, hardware specific tests are needed and they must be run with the hardware available. This directory provides these tests. For now the only hardware specific tests are for the Pi Camera. They must be run on a Raspberry Pi with a Pi Camera attached.
+
+In GitLab we track our "code coverage", this tells us which lines of code have been executed during testing. The `picamera_tests.py` script in the root of the repository can be used to report the coverage from these tests. See the section below on "Reporting the coverage to GitLab".
+
+### Running and reporting the test coverage to GitLab (for merge requests)
+
+It is essential to stop the server before running these tests:
+
+    ofm stop
+
+To create a coverage report that will be included into the repository (and reported to GitLab) run:
+
+    ./picamera_tests.py run
+
+This will first run `pytest` on the hardware specific tests. This creates a `.coverage` database that can be used to analyse the code coverage of these tests. This will be copied to `.coverage.picamera`. The script will then create a zip of these test results along with hashes of source code for the Pi Camera:
+
+* `picamera.py`
+* `picamera_recalibrate_utils.py`
+* `__init__.py` in the `camera` directory that defines `BaseCamera`
+
+This zip should then be committed to the repository.
+
+### Running these tests during development
+
+As before, the server must be stopped before running these tests.
+
+The camera test are very slow as they run on hardware. They can be run with:
+
+    pytest tests/hardware_specific_tests
+
+However, this will not archive the tests in the Git repository for reporting the coverage. For this, see the section above on reporting the coverage.
+
+When writing and debugging these unit tests, it is often best to run a specific test and to use the `-s` flag to see the print statements. It is also often useful to use `--pdb` to drop you into a python debug session on any failure. For example, you might run:
+
+    pytest tests/hardware_specific_tests/picamera2/test_exposure_time_drift.py::test_exposure_time_saves_and_loads -s --pdb
+
+### CI explanation
+
+When the CI runs on GitLab to calculate code coverage, it will first run the tests in the `tests` directory in one job, and archive this as `.coverage.main`. Then it will run a second job that:
+
+* Imports the archive of `.coverage.main` for the tests just run on the server
+* Unzip the zip of the results of running tests on the Pi Camera (`.coverage.picamera`)
+* Check that the hashes for the Pi Camera source code have not changed. If they have changed it will error and ask for the picamera tests to be re-run on a Raspberry Pi.
+* Runs `coverage combine` to create a single `.coverage` report that combines the coverage from both `.coverage.main` and `.coverage.picamera`.
+* Generates the information needed to display the coverage in GitLab
+
+This ensures that:
+
+* Hardware specific tests are re-run if the relevant source code changes.
+* The coverage for hardware specific tests is reported correctly
+* The hardware specific tests do not need re-running when other code that does not affect PiCamera interaction is updated.
+
+### Keeping your branch up to date
+
+One of the downsides of this form of testing is that if both the target and the source branch have changes to the camera files, then the merged result will fail on CI. This is because the tests were not run against the merged result.
+
+As such, any branches making camera changes need to kept in sync with their target branch. It is preferable to do this with rebasing, to prevent a very confusing series of merge commits.
+
+To rebase your branch, first make sure it is checked out by running (with BRANCH_NAME updated):
+
+`git checkout BRANCH_NAME`
+
+Make sure your computer knows the server's current state by fetching:
+
+`git fetch`
+
+Then run the rebase (assuming `v3` is the target branch):
+
+`git rebase origin/v3`
+
+**Note:** If both branches have updated the zip, you may have a `both modified` conflict on the zip itself. The best way to resolve this is to just to use the file from the branch you are rebasing on. You can do this by running the slightly confusing command of:
+
+`git checkout --ours -- picamera_coverage.zip`
+
+if you then run:
+
+`git add -A`  
+`git status`
+
+You should see that there is now no conflict and that `picamera_coverage.zip` is not listed as modified at all. To carry on rebasing run:
+
+`git rebase --continue`
+
+Once done you can push your rebased changes with:
+
+`git push --force-with-lease`
+
+Do **not** just run `git push` and then follow the instructions to first "pull". This can create a horrible mess.
+
+
+**If you want to use a branch that has been rebased on another computer.**
+
+If you try to pull this branch on another computer that already had an older version checked out, you will get a warning about the force push. In this case, we want the server's history exactly and want to forget the history on that computer.
+
+**First:** Check you have no unsaved work on this branch, as we are about to delete it!
+
+Then you can replace the local history by running (with BRANCH_NAME updated):
+
+`git reset --hard origin/BRANCH_NAME`
+
+
+### Creating a combined report locally
+
+To create a combined coverage report locally (similar to the one created on CI) first run all normal tests with `pytest`.
+
+Copy the `.coverage` file to `.coverage.main`
+
+    cp .coverage .coverage.main
+
+Then run:
+
+    ./picamera_tests.py unzip
+
+to get the `.coverage.picamera` file.
+
+To combine reports run:
+
+    coverage combine
+
+It is then possible to run:
+
+* `coverage report` to get the in-terminal report of the coverage.
+* `coverage html` to get the HTML overview of covered lines.
+* `coverage xml` to get an XML report any software that may need it.

tests/hardware_specific_tests/picamera2/cam_test_utils.py

@@ -0,0 +1,37 @@
+"""Utilities to help with testing the camera."""
+
+from contextlib import contextmanager
+from typing import Optional
+
+from openflexure_microscope_server.things.camera.picamera import StreamingPiCamera2
+
+from ...shared_utils.lt_test_utils import LabThingsTestEnv
+
+
+@contextmanager
+def camera_test_env(settings_folder: Optional[str] = None):
+    """Yield a test environment with a server that contains just the camera.
+
+    This is a context manager, not a pytest fixture, as it needs to be created
+    multiple times in some tests.
+
+    :param settings_folder: The settings folder for the camera, if none is supplied, new
+        temporary directory will be used as the settings folder.
+    """
+    thing_conf = {"camera": StreamingPiCamera2}
+    with LabThingsTestEnv(things=thing_conf, settings_folder=settings_folder) as env:
+        yield env
+
+
+@contextmanager
+def camera_test_client(settings_folder: Optional[str] = None):
+    """Yield a camera ThingClient on a camera server.
+
+    This is a context manager, not a pytest fixture, as it needs to be created
+    multiple times in some tests.
+
+    :param settings_folder: The settings folder for the camera, if none is supplied, new
+        temporary directory will be used as the settings folder.
+    """
+    with camera_test_env(settings_folder=settings_folder) as env:
+        return env.get_thing_client("camera")

tests/hardware_specific_tests/picamera2/conftest.py

@@ -0,0 +1,21 @@
+"""Shared fixtures for picamera tests."""
+
+import pytest
+
+import labthings_fastapi as lt
+
+from .cam_test_utils import camera_test_client_and_server
+
+
+@pytest.fixture
+def picamera_test_env() -> lt.ThingClient:
+    """Initialise a test environment with only a StreamingPiCamera2 Thing."""
+    with camera_test_client_and_server() as env:
+        yield env
+
+
+@pytest.fixture
+def picamera_client() -> lt.ThingClient:
+    """Initialise a test picamera_client (in a LabThings test env)."""
+    with camera_test_client_and_server() as env:
+        return env.get_thing_client["camera"]

tests/hardware_specific_tests/picamera2/test_acquisition.py

@@ -0,0 +1,31 @@
+"""Test data collection from the Raspberry Picamera."""
+
+import numpy as np
+from PIL import Image
+
+
+def test_jpeg_and_array(picamera_client):
+    """Check that a jpeg grabbed from the stream is the same size as other captures.
+
+    Compare it to an array capture and a jpeg capture.
+    """
+    # Grab a jpeg from the stream
+    blob = picamera_client.grab_jpeg()
+    mjpeg_frame = Image.open(blob.open())
+    # Verify throws an error if there are issues with the image
+    mjpeg_frame.verify()
+    assert mjpeg_frame.format == "JPEG"
+
+    # Capture a jpeg
+    blob = picamera_client.capture_jpeg(stream_name="main")
+    jpeg_capture = Image.open(blob.open())
+    jpeg_capture.verify()
+    assert jpeg_capture.format == "JPEG"
+
+    # Capture an array
+    arrlist = picamera_client.capture_array(stream_name="main")
+    array_main = np.array(arrlist)
+
+    # Verify image sizes are the same
+    assert mjpeg_frame.size == jpeg_capture.size
+    assert array_main.shape[1::-1] == jpeg_capture.size

tests/hardware_specific_tests/picamera2/test_calibration.py

@@ -0,0 +1,72 @@
+"""Test data collection from the Raspberry Picamera."""
+
+import tempfile
+from copy import deepcopy
+
+from openflexure_microscope_server.things.camera.picamera import StreamingPiCamera2
+
+from .cam_test_utils import camera_test_client
+
+
+def test_calibration(picamera_test_env):
+    """Check that full auto calibrate completes and set the expected values."""
+    picamera_client, server = picamera_test_env.get_thing_client["camera"]
+    picamera_thing = picamera_test_env.get_thing_by_type(StreamingPiCamera2)
+    # Check the calibration_required property used by the calibration wizard
+    assert picamera_thing.calibration_required
+    # Save copy of default tuning file for end of test
+    original_default = deepcopy(picamera_thing.default_tuning)
+    # Tuning should start the same as the server is loading with no settings.
+    assert picamera_thing.default_tuning == picamera_thing.tuning
+    # The background detector isn't ready as there is no background image.
+    assert not picamera_thing.background_detector_status.ready
+
+    # Run full auto calibrate
+    picamera_client.full_auto_calibrate()
+
+    # After calibration it should report that calibration is no longer  required
+    assert not picamera_thing.calibration_required
+    # The tuning files should be different
+    assert picamera_thing.default_tuning != picamera_thing.tuning
+    # The default should be unchanged
+    assert picamera_thing.default_tuning == original_default
+
+    assert picamera_thing.background_detector_status.ready
+
+
+def test_tuning_is_persistent():
+    """Check that tuning file adjustments are persistent."""
+    # First check full auto-calibrate is persistent if same save dir is used.
+    with tempfile.TemporaryDirectory() as tmpdir:
+        with camera_test_client(settings_folder=tmpdir) as client:
+            initial_tuning = deepcopy(client.tuning)
+            client.full_auto_calibrate()
+            calibrated_tuning = deepcopy(client.tuning)
+            # Full auto calibrate should have saved the tuning.
+            assert initial_tuning != calibrated_tuning
+
+        # Reload the camera and server from scratch
+        with camera_test_client(settings_folder=tmpdir) as client:
+            initial_tuning = deepcopy(client.tuning)
+            # On reload the initial tuning is as it was when calibrated last run
+            assert initial_tuning == calibrated_tuning
+            # Set the lens shading to be flat
+            client.flat_lens_shading()
+            flat_tuning = deepcopy(client.tuning)
+            assert flat_tuning != calibrated_tuning
+
+        # Reload to check the flat lst persists
+        with camera_test_client(settings_folder=tmpdir) as client:
+            initial_tuning = deepcopy(client.tuning)
+            # On reload the initial tuning the flat tuning
+            assert initial_tuning == flat_tuning
+            # Reset the lens shading to what is in the default tuning file
+            client.reset_lens_shading()
+            flat_tuning = deepcopy(client.tuning)
+            assert flat_tuning != calibrated_tuning
+
+        # Reload the one more time to check the reset lst persists
+        with camera_test_client(settings_folder=tmpdir) as client:
+            initial_tuning = deepcopy(client.tuning)
+            # On reload the initial tuning the flat tuning
+            assert initial_tuning == flat_tuning

tests/hardware_specific_tests/picamera2/test_exposure_time_drift.py

@@ -0,0 +1,176 @@
+"""Check exposure times do not drift.
+
+This can get very tedious. Recommend running pytest with -s option
+to monitor progress.
+"""
+
+import json
+import logging
+import os
+import tempfile
+import time
+from typing import Any
+
+from .cam_test_utils import camera_test_client
+
+logging.basicConfig(level=logging.DEBUG)
+
+# The tolerance used to be a setting of the camera, and any changes within tolerance
+# were ignored. Now this isn't needed we always set a value 1 larger than the last
+# accepted value as the PiCamera always rounds down.
+# The tolerance is needed in this test as when setting an arbitrary value it is rounded
+# down to a hardware compatible one.
+EXPOSURE_TOL = 30
+
+
+def _test_exposure_time_drift(desired_time: int) -> None:
+    """Capture 10 full res images and check that the exposure time remains constant.
+
+    This confirms that automatic exposure time adjustment is fully turned off during
+    capture.
+    """
+    with camera_test_client() as client:
+        client.exposure_time = desired_time
+        print(f"Setting desired time of {desired_time}")
+        time.sleep(0.5)
+        pre_capture_et = client.exposure_time
+        print(f"Pre-capture the time is set to {pre_capture_et}")
+        # Check exp is set correctly within known tolerance
+        assert abs(pre_capture_et - desired_time) < EXPOSURE_TOL
+
+        for i in range(10):
+            client.capture_jpeg(stream_name="full")
+            if i == 0:
+                # Exposure can update on first capture, due to frame rate restrictions
+                first_et = client.exposure_time
+                assert abs(first_et - desired_time) < EXPOSURE_TOL
+            else:
+                frame_et = client.exposure_time
+                print(f"Frame {i} captured with exposure time {frame_et}")
+                # Check no further drift in value
+                assert first_et == frame_et
+
+        # Set the exposure time to the value it already is. To check it doesn't shift
+        print(f"Setting exposure time to {frame_et} to check it doesn't change")
+        client.exposure_time = frame_et
+        time.sleep(0.5)
+        # Check before and after capture
+        assert client.exposure_time == frame_et
+        client.capture_jpeg(stream_name="full")
+        assert client.exposure_time == frame_et
+        print("Exposure time didn't change!!")
+    print(f"End of test for exposure target {desired_time}")
+
+
+def test_exposure_time_drift():
+    """Performs the exposure time test for a range of exposure time values."""
+    for desired_time in [100, 1000, 10000]:
+        _test_exposure_time_drift(desired_time)
+
+
+def test_exposure_time_on_start_and_stop_stream():
+    """Start and stop stream in the same way a scan does and check exposure doesn't drift.
+
+    Take images using capture_to_memory() just as a scan would.
+    """
+    desired_time = 1000
+    with camera_test_client() as client:
+        # Set a desired exposure time
+        client.exposure_time = desired_time
+        print(f"Setting desired time of {desired_time}")
+        time.sleep(0.5)
+        # Take a couple of images to make sure that the exposure is adjusted to
+        # a hardware compatible value.
+        for _i in range(2):
+            client.capture_jpeg(stream_name="full")
+        # Save this time.
+        set_time = client.exposure_time
+        assert abs(set_time - desired_time) < EXPOSURE_TOL
+
+        # Mimic doing 10 smart scans. Change to full res, take some images. Return
+        # to standard preview resolution.
+        for i in range(10):
+            print(f"Starting simulation scan {i}")
+            # This will need updating if we start supporting other Picamera models
+            # It is currently used here to mimic the behaviour in in scanning.
+            client.start_streaming(main_resolution=(3280, 2464))
+            time.sleep(0.5)
+            for _j in range(5):
+                client.capture_to_memory(buffer_max=1)
+            # Reset to main resolution
+            time.sleep(0.5)
+            client.start_streaming()
+        # Check after all of this the exposure time is the same.
+        assert client.exposure_time == set_time
+
+
+def _load_camera_and_return_exposure(tmpdir: str) -> int:
+    """Load a camera (using any settings in tempdir) take images and check exposure."""
+    with camera_test_client(settings_folder=tmpdir) as client:
+        # Set a desired exposure time
+        time.sleep(0.5)
+        # Take a couple of images to make sure that the exposure is adjusted to
+        # a hardware compatible value.
+        for _i in range(2):
+            client.capture_jpeg(stream_name="full")
+        # Save this time.
+        return client.exposure_time
+
+
+def _load_setting(setting_file: str) -> dict[str, Any]:
+    """Load settings json from disk to dictionary."""
+    with open(setting_file, "r", encoding="utf-8") as file_obj:
+        return json.load(file_obj)
+
+
+def _save_setting(settings: dict[str, Any], setting_file: str):
+    """Save settings dictionary to disk."""
+    with open(setting_file, "w", encoding="utf-8") as file_obj:
+        json.dump(settings, file_obj)
+
+
+def test_exposure_time_saves_and_loads():
+    """Check that exposure time saves to disk and loads correctly."""
+    with tempfile.TemporaryDirectory() as tmpdir:
+        setting_file = os.path.join(tmpdir, "camera", "settings.json")
+
+        # Create a server, take some images, and get the exposure time
+        initial_exposure = _load_camera_and_return_exposure(tmpdir)
+        settings = _load_setting(setting_file)
+        assert settings["exposure_time"] == initial_exposure
+
+        # Adjust exposure to 1000 and save
+        settings["exposure_time"] = 1000
+        _save_setting(settings, setting_file)
+
+        # Create a server, take some images, and get the exposure time
+        recorded_exposure = _load_camera_and_return_exposure(tmpdir)
+        settings = _load_setting(setting_file)
+        assert settings["exposure_time"] == recorded_exposure
+        # Check it was set correctly within tolerance
+        assert abs(recorded_exposure - 1000) < EXPOSURE_TOL
+
+        # Load a second time without changing the file. Exposure should not change
+        recorded_exposure_second_load = _load_camera_and_return_exposure(tmpdir)
+        settings = _load_setting(setting_file)
+        assert settings["exposure_time"] == recorded_exposure_second_load
+        # Check it was set to exactly the value previously saved to disk
+        assert recorded_exposure == recorded_exposure_second_load
+
+        # Repeat with 2000
+        settings["exposure_time"] = 2000
+        _save_setting(settings, setting_file)
+
+        # Create a server, take some images, and get the exposure time
+        recorded_exposure = _load_camera_and_return_exposure(tmpdir)
+        settings = _load_setting(setting_file)
+        assert settings["exposure_time"] == recorded_exposure
+        # Check it was set correctly within tolerance
+        assert abs(recorded_exposure - 2000) < EXPOSURE_TOL
+
+        # Load a second time without changing the file. Exposure should not change
+        recorded_exposure_second_load = _load_camera_and_return_exposure(tmpdir)
+        settings = _load_setting(setting_file)
+        assert settings["exposure_time"] == recorded_exposure_second_load
+        # Check it was set to exactly the value previously saved to disk
+        assert recorded_exposure == recorded_exposure_second_load

tests/hardware_specific_tests/picamera2/test_sensor_mode.py

@@ -0,0 +1,21 @@
+"""Test changing and setting camerat modes on the Raspberry Picamera."""
+
+import logging
+
+import numpy as np
+
+from .cam_test_utils import camera_test_client
+
+logging.basicConfig(level=logging.DEBUG)
+
+
+def test_sensor_mode():
+    """Test capturing raw arrays in two different sensor modes."""
+    with camera_test_client() as client:
+        for size in [(3280, 2464), (1640, 1232)]:
+            client.sensor_mode = {"output_size": size, "bit_depth": 10}
+            arr = np.array(client.capture_array(stream_name="raw"))
+            # Check that the array dimensions match the requested image size.
+            # Note: Numpy array shape is (y,x), but the sensor is set with (x,y)
+            # hence the need to compare index 0 with index 1.
+            assert arr.shape[0] == size[1]

tests/hardware_specific_tests/picamera2/test_tuning.py

@@ -0,0 +1,100 @@
+"""Tests that check that a tuning file can be reloaded."""
+
+import pytest
+from picamera2 import Picamera2
+
+from openflexure_microscope_server.things.camera import (
+    picamera_recalibrate_utils as recalibrate_utils,
+)
+from openflexure_microscope_server.things.camera import (
+    picamera_tuning_file_utils as tf_utils,
+)
+
+MODEL = Picamera2.global_camera_info()[0]["Model"]
+
+
+def generate_bad_tuning():
+    """Return a tuning file with an invalid version number to force an error when loaded."""
+    default_tuning = tf_utils.load_default_tuning("imx219")
+    bad_tuning = default_tuning.copy()
+    bad_tuning["version"] = 999
+    return bad_tuning
+
+
+def _test_bad_tuning_after_good_tuning(configure: bool = False):
+    """Test loading good, bad, then another good tuning files in sequence.
+
+    Load the default tuning file into the camera, re-load with a broken tuning file,
+    check it errors. Finally check the default tuning file will load again afterwards.
+
+    :param configure: Boolean, set true to configure the camera on initial loading
+
+    This test checks that:
+    recalibrate_utils.recreate_camera_manager() is working as expected. As the default
+    PiCamera2 behaviour does not expect the tuning file to be reloaded.
+    """
+    bad_tuning = generate_bad_tuning()
+    default_tuning = tf_utils.load_default_tuning("imx219")
+    print("opening camera with default tuning")
+    with Picamera2(tuning=default_tuning) as cam:
+        if configure:
+            cam.configure(cam.create_preview_configuration())
+    del cam
+    recalibrate_utils.recreate_camera_manager()
+    print(f"Opening camera with bad tuning - ['version'] = {bad_tuning['version']}")
+    with pytest.raises(IndexError):
+        # The bad version should cause a problem
+        cam = Picamera2(tuning=bad_tuning)
+    recalibrate_utils.recreate_camera_manager()
+    with Picamera2(tuning=default_tuning) as cam:
+        # Reload the camera with working tuning, or it will stop responding
+        # and fail future tests
+        pass
+    del cam
+
+
+# Note: The tests below are marked with
+# @pytest.mark.filterwarnings("ignore: Exception ignored")
+# as the picamera will throw an exception ignored warning when deleting
+# the picamera object. This is expected
+
+# The 2 pairs of repeated identical tests exist because pytest only
+# starts once, and so by running the test more than one time checks
+# that the camera can be initialised multiple times without restarting
+# python.
+
+
+@pytest.mark.filterwarnings("ignore: Exception ignored")
+def test_bad_tuning_after_good_tuning_noconfigure():
+    """First run setting good, then bad, then good tuning.
+
+    create_preview_configuration() is NOT run on initial setup.
+    """
+    _test_bad_tuning_after_good_tuning(configure=False)
+
+
+@pytest.mark.filterwarnings("ignore: Exception ignored")
+def test_bad_tuning_after_good_tuning_configure():
+    """Second run setting good, then bad, then good tuning.
+
+    create_preview_configuration() is run on initial setup this time.
+    """
+    _test_bad_tuning_after_good_tuning(configure=True)
+
+
+@pytest.mark.filterwarnings("ignore: Exception ignored")
+def test_bad_tuning_after_good_tuning_noconfigure2():
+    """3rd run setting good, then bad, then good tuning.
+
+    create_preview_configuration() is NOT run on initial setup.
+    """
+    _test_bad_tuning_after_good_tuning(configure=False)
+
+
+@pytest.mark.filterwarnings("ignore: Exception ignored")
+def test_bad_tuning_after_good_tuning_configure2():
+    """Final run setting good, then bad, then good tuning.
+
+    create_preview_configuration() is run on initial setup again.
+    """
+    _test_bad_tuning_after_good_tuning(configure=True)

tests/unit_tests/__init__.py

@@ -1,6 +1,6 @@
 """The unit-test suite for the OpenFlexure Microscope Server.
 
 This package contains all of the unit tests that can be run without specific
-hardware. See also the `hardware-specific-tests` directory and the
+hardware. See also the `hardware_specific_tests` directory and the
 `integration-tests` directory for more testing!.
 """

tests/unit_tests/test_cameras.py

@@ -1,6 +1,6 @@
 """Tests for camera classes. These tests focus on checking the camera APIs are equivalent.
 
-Tests for specific camera hardware are in the hardware-specific-tests directory. Tests
+Tests for specific camera hardware are in the hardware_specific_tests directory. Tests
 on camera functionality using the simulation camera are in "test_camera".
 """