Skip to content

Fix integration test harness #165

Description

@jbriones1

Description

Our test harness is very janky and it should be fixed before testing can be enforced properly. I've been band-aiding our test harness, but I think it needs a full touch-up before more people contribute and the bad patterns I've introduced should be fixed.

Issues

  • Reliance on seeding the database multiple times when running a full test suite.
  • Reliance on mock data being actually inserted into the database.
  • Test effects leak between tests, so test order can change results.
  • Database sessions sometimes used the one defined in integration/conftest.py, but sometimes they use the one defined for the application.
  • We have markers declared in pyproject.toml, but they're not used.
  • The way fixtures are scoped need to be made more exact.

Proper test flow

Session scoped

  1. Application started in test mode.
  2. Test database's schema is prepared and the database is emptied.

Function scoped

  1. Each test creates a database transaction/session.
  2. Each test will seed their transaction with data.
  3. Clients are created per test.
  4. Test runs.
  5. Test effects are cleaned up.

Changes needed

This only applies to the integration test suite, the unit tests will be looked at another time.

Loading the Test Database

  • Create function-scoped data to load for tests instead of using load_test_db.py.
  • Clean up the test data to be more generic i.e. remove references to real people.
  • Seed the database with a generic session that's active at the time of insertion.

Fixtures

  • Change the fixture hierarchy to:
test_database        session/module scope
└── db_connection    function scope + outer transaction
    ├── db_session   function scope
    └── client       function scope
        └── FastAPI creates sessions using db_connection
  • Database creation needs to always target the test database and starts with an empty database.
  • Database sessions need to create connections that won't commit their changes to the real database using join_transaction_mode=create_savepoint.
  • Transactions need to be rolled back when they complete.
  • Create mock data factories on fixtures to seed a transaction before a test instead of relying on load_test_db.py.
    • These factories need to use db_session.flush() instead of db_session.commit().
  • Mark tests with pytestmark = [pytest.mark.integration, pytest.mark.asyncio(loop_scope="session")] at the module level.
    • This will allow us to call integration tests with pytest tests/integration -m integration

Clients

These are used to emulate users contacting the web server.

  • Make the clients function scoped so a fresh cookie and headers are set for each test.
  • Admin client seeds the database with a session and any supporting entries.
  • Create an unauthorized client
  • Create a client with election permissions

Tests

We'll need to check our test suite and update them.

  • Authentication
  • Elections
  • Image Asset
  • Nominees
  • Officers
  • Honorary Members
  • Candidates
  • Event
  • TransLink

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingenhancementNew feature or request

    Type

    Projects

    Status
    Ready

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions