Testing Guide

Learn how to write and run tests in CVAT

Testing is crucial for maintaining CVAT’s quality and reliability. This guide covers how to run existing tests and write new ones.

Test Structure

CVAT has several test suites:

• Python REST API tests: tests/python/rest_api/
• Python SDK tests: tests/python/sdk/
• Python CLI tests: tests/python/cli/
• Frontend unit tests: In component directories
• E2E tests: tests/cypress/e2e/

Prerequisites

Before running tests:

1. Ensure Docker is running
2. Stop any existing CVAT instances that might use the same ports
3. Install Python dependencies:

pip install -r tests/python/requirements.txt

4. Install Node.js dependencies:

cd tests
yarn install

Running Python Tests

REST API Tests

The REST API tests use pytest and run against a real CVAT instance in Docker containers.

Run All Tests

pytest tests/python/

This command automatically:

• Starts all necessary Docker containers
• Runs the complete test suite
• Cleans up after completion

Run Specific Test Files

pytest tests/python/rest_api/test_tasks.py

Run Specific Test Functions

pytest tests/python/rest_api/test_tasks.py::test_create_task

Run with Verbose Output

pytest tests/python/ -v

Run Tests in Parallel

pytest tests/python/ -n auto

SDK Tests

pytest tests/python/sdk/

CLI Tests

pytest tests/python/cli/

Keep Test Services Running

To keep the test environment running for debugging:

pytest tests/python/ --start-services

This starts the containers but doesn’t run tests, allowing you to:

• Access the test instance at http://localhost:8080
• Inspect the database
• Test API endpoints manually

Stop the services:

pytest tests/python/ --stop-services

Running Frontend Tests

Unit Tests

CVAT uses Jest for frontend unit testing:

cd cvat-ui
yarn test

Run with coverage:

yarn test --coverage

Run specific test files:

yarn test TaskItem.test.tsx

Run in watch mode:

yarn test --watch

Running E2E Tests

E2E tests use Cypress to test the complete application flow.

Interactive Mode

Open Cypress Test Runner:

cd tests
yarn run cypress:open

This opens an interactive interface where you can:

• Select specific tests to run
• Watch tests execute in real-time
• Debug failing tests
• See screenshots and videos

Headless Mode

Run all E2E tests in headless mode:

cd tests
yarn run cypress:run

Run specific test file:

yarn run cypress:run --spec "cypress/e2e/actions_tasks.js"

E2E Test Configuration

Cypress configuration is in tests/cypress.config.js. Common settings:

{
  baseUrl: 'http://localhost:8080',
  video: true,
  screenshotOnRunFailure: true,
  defaultCommandTimeout: 10000
}

Writing Tests

Writing Python REST API Tests

REST API tests are located in tests/python/rest_api/.

Test Structure

import pytest
from http import HTTPStatus

class TestTasks:
    def test_create_task(self, admin_user):
        """Test creating a new task."""
        # Arrange
        task_spec = {
            "name": "Test Task",
            "labels": [{"name": "car"}]
        }
        
        # Act
        response = admin_user.post(
            "/api/tasks",
            json=task_spec
        )
        
        # Assert
        assert response.status_code == HTTPStatus.CREATED
        assert response.json()["name"] == "Test Task"
        assert len(response.json()["labels"]) == 1

    def test_get_task(self, admin_user, tasks):
        """Test retrieving a task by ID."""
        task_id = tasks[0]["id"]
        
        response = admin_user.get(f"/api/tasks/{task_id}")
        
        assert response.status_code == HTTPStatus.OK
        assert response.json()["id"] == task_id

Using Fixtures

Common fixtures are defined in tests/python/shared/fixtures/:

@pytest.fixture
def admin_user():
    """Fixture providing an authenticated admin user."""
    # Returns a test client authenticated as admin
    pass

@pytest.fixture
def tasks():
    """Fixture providing test tasks."""
    # Returns list of test tasks
    pass

Writing Frontend Unit Tests

Frontend tests use Jest and React Testing Library.

Component Test Example

import { render, screen, fireEvent } from '@testing-library/react';
import TaskItem from './TaskItem';

describe('<TaskItem />', () => {
    const mockTask = {
        id: 1,
        name: 'Test Task',
        status: 'annotation',
    };
    
    const mockOnSelect = jest.fn();
    
    beforeEach(() => {
        mockOnSelect.mockClear();
    });
    
    it('renders task name', () => {
        render(<TaskItem task={mockTask} onSelect={mockOnSelect} />);
        expect(screen.getByText('Test Task')).toBeInTheDocument();
    });
    
    it('calls onSelect when clicked', () => {
        render(<TaskItem task={mockTask} onSelect={mockOnSelect} />);
        fireEvent.click(screen.getByText('Test Task'));
        expect(mockOnSelect).toHaveBeenCalledWith(1);
    });
    
    it('displays task status', () => {
        render(<TaskItem task={mockTask} onSelect={mockOnSelect} />);
        expect(screen.getByText('annotation')).toBeInTheDocument();
    });
});

Writing E2E Tests

E2E tests are in tests/cypress/e2e/.

Cypress Test Example

describe('Task creation', () => {
    beforeEach(() => {
        cy.visit('/');
        cy.login('admin1', 'password');
    });
    
    it('creates a new task', () => {
        // Navigate to task creation
        cy.contains('Create new task').click();
        
        // Fill in task details
        cy.get('[placeholder="Task name"]').type('My Test Task');
        cy.contains('Add label').click();
        cy.get('[placeholder="Label name"]').type('car');
        
        // Upload images
        cy.get('input[type="file"]').attachFile('sample_image.jpg');
        
        // Submit
        cy.contains('Submit').click();
        
        // Verify task was created
        cy.contains('Task has been created').should('be.visible');
        cy.contains('My Test Task').should('be.visible');
    });
});

Test Database

The test infrastructure uses a pre-populated database.

Database Reset

The database is automatically restored after each test function to ensure test isolation.

Updating Test Database

If you need to update the test database:

1. Start test services:

   pytest tests/python/ --start-services
   

2. Make changes through the UI or API

3. Backup the database:

   cd tests/python
   docker exec test_cvat_server_1 python manage.py dumpdata --indent 2 --natural-foreign \
       --exclude=admin --exclude=auth.permission --exclude=authtoken --exclude=contenttypes \
       --exclude=django_rq --exclude=sessions \
       > shared/assets/cvat_db/data.json
   

4. Backup data volume:

   docker exec test_cvat_server_1 tar --exclude "/home/django/data/cache" -cjv /home/django/data \
       > shared/assets/cvat_db/cvat_data.tar.bz2
   

5. Update JSON assets:

   python shared/utils/dump_objects.py
   

Test Coverage

Check test coverage for Python:

pytest tests/python/ --cov=cvat --cov-report=html

View coverage report:

open htmlcov/index.html

For frontend coverage:

cd cvat-ui
yarn test --coverage

Debugging Tests

Python Tests

Add breakpoints:

import pdb; pdb.set_trace()

Run with debug output:

pytest tests/python/ -vv -s

Frontend Tests

Run in watch mode:

yarn test --watch

Use console.log or debugger statements.

Cypress Tests

Use interactive mode:

yarn run cypress:open

Add Cypress debug commands:

cy.debug();
cy.pause();

Continuous Integration

CVAT’s CI automatically runs:

• All Python tests
• Frontend unit tests
• E2E tests
• Linters and type checks
• Security scans

Tests must pass before PRs can be merged.

Troubleshooting

Port Conflicts

If test containers fail to start due to port conflicts:

docker ps -a
docker stop $(docker ps -a -q)

Database Issues

Recreate the test database:

docker exec test_cvat_db_1 dropdb --if-exists cvat
docker exec test_cvat_db_1 createdb cvat
docker exec test_cvat_server_1 python manage.py migrate

Test Failures

For date/time related failures, update JSON assets:

python tests/python/shared/utils/dump_objects.py

Best Practices

1. Write isolated tests: Each test should be independent
2. Use descriptive names: Test names should clearly describe what they test
3. Follow AAA pattern: Arrange, Act, Assert
4. Test edge cases: Include tests for error conditions
5. Keep tests fast: Mock external dependencies when possible
6. Maintain test data: Keep test fixtures up to date
7. Clean up: Ensure tests don’t leave side effects

Next Steps

• Review the architecture documentation
• Learn about the pull request process
• Check the code style guidelines