Testing Infrastructure

The following files were used as context for generating this wiki page:

• .gitignore
• LICENSE
• setup.py

Purpose and Scope

This document describes the testing infrastructure for WSHawk, including test frameworks, test organization, coverage measurement, and test execution workflows. It is intended for developers contributing to WSHawk or extending its functionality with new features.

For information about building WSHawk from source, see 11.3. For the development environment setup, see 11.1. For Docker-based development workflows, see 11.8.

---

Testing Stack Overview

WSHawk's testing infrastructure is built on industry-standard Python testing tools, supporting multiple Python versions (3.8-3.13) and enabling comprehensive verification of functionality across different environments.

graph TB
    subgraph "Test Execution Layer"
        pytest["pytest<br/>Test Runner<br/>Fixtures & Parametrization"]
        tox["tox<br/>Multi-Environment Testing<br/>Python 3.8-3.13"]
    end
    
    subgraph "Test Analysis Layer"
        coverage["coverage.py<br/>Code Coverage<br/>Branch Analysis"]
        hypothesis["hypothesis<br/>Property-Based Testing<br/>Fuzz Generation"]
    end
    
    subgraph "Test Organization"
        unit["Unit Tests<br/>tests/unit/<br/>Scanner, Payloads, Utils"]
        integration["Integration Tests<br/>tests/integration/<br/>End-to-End Scans"]
        fixtures["Test Fixtures<br/>tests/fixtures/<br/>Mock Servers, Payloads"]
    end
    
    subgraph "Test Artifacts"
        cache[".pytest_cache/<br/>Test Session Cache"]
        covdata[".coverage<br/>Coverage Data File"]
        htmlcov["htmlcov/<br/>HTML Coverage Report"]
        toxenv[".tox/<br/>Virtual Environments"]
        hypdata[".hypothesis/<br/>Example Database"]
    end
    
    subgraph "CI/CD Integration"
        ghactions["GitHub Actions<br/>.github/workflows/<br/>test.yml"]
    end
    
    pytest --> unit
    pytest --> integration
    pytest --> fixtures
    tox --> pytest
    
    pytest --> coverage
    pytest --> hypothesis
    
    pytest --> cache
    coverage --> covdata
    coverage --> htmlcov
    tox --> toxenv
    hypothesis --> hypdata
    
    ghactions --> tox
    ghactions --> coverage
    
    fixtures --> unit
    fixtures --> integration

Sources: setup.py:24, .gitignore:38-43

---

Test Organization Structure

WSHawk follows a hierarchical test organization pattern, with tests excluded from the distributed package.

Directory Layout

wshawk/
├── tests/                  # Root test directory (excluded from package)
│   ├── __init__.py
│   ├── unit/              # Unit tests for individual modules
│   │   ├── test_scanner_v2.py
│   │   ├── test_payloads.py
│   │   ├── test_resilience.py
│   │   ├── test_analysis.py
│   │   └── test_integrations.py
│   ├── integration/       # End-to-end integration tests
│   │   ├── test_full_scan.py
│   │   ├── test_web_dashboard.py
│   │   └── test_cli.py
│   ├── fixtures/          # Shared test fixtures and data
│   │   ├── mock_servers.py
│   │   ├── sample_payloads.json
│   │   └── vulnerable_endpoints.py
│   └── conftest.py        # pytest configuration and shared fixtures
├── examples/              # Example scripts (excluded from package)
└── docs/                  # Documentation (excluded from package)

The exclude directive in setup.py ensures that test code, examples, and documentation are not included in the PyPI distribution package, reducing package size and avoiding namespace pollution.

Sources: setup.py:24

Test Naming Conventions

| Test Type | File Pattern | Class Pattern | Function Pattern | |-----------|--------------|---------------|------------------| | Unit | test_<module>.py | Test<Class> | test_<function>_<scenario> | | Integration | test_<feature>.py | TestIntegration<Feature> | test_<workflow>_<outcome> | | Property | test_<module>_properties.py | TestProperties | test_property_<invariant> |

---

Test Execution Framework

pytest Configuration

The primary test runner is pytest, which provides powerful fixtures, parametrization, and plugin support. A typical conftest.py configuration includes:

# tests/conftest.py (typical structure)
import pytest
from wshawk.scanner_v2 import WSHawkV2
from wshawk.payloads import WSPayloads

@pytest.fixture
def scanner_instance():
    """Provides a configured WSHawkV2 instance for testing."""
    scanner = WSHawkV2("ws://test.example.com")
    scanner.use_headless_browser = False  # Disable for faster tests
    scanner.use_oast = False
    return scanner

@pytest.fixture
def payload_collection():
    """Provides WSPayloads instance with test payloads."""
    return WSPayloads()

@pytest.fixture(scope="session")
def mock_vulnerable_server():
    """Starts a mock vulnerable WebSocket server for integration tests."""
    # Start server, yield port, teardown
    pass

Running Tests Locally

Basic Test Execution:

# Run all tests
pytest

# Run with verbose output
pytest -v

# Run specific test file
pytest tests/unit/test_scanner_v2.py

# Run specific test function
pytest tests/unit/test_scanner_v2.py::TestScanner::test_injection_detection

# Run tests matching pattern
pytest -k "injection"

# Run with print statements visible
pytest -s

Parallel Execution:

# Install pytest-xdist
pip install pytest-xdist

# Run tests in parallel (4 workers)
pytest -n 4

---

Coverage Measurement

WSHawk uses coverage.py to measure code coverage and ensure comprehensive testing of critical security functionality.

graph LR
    tests["Test Suite<br/>pytest execution"]
    covrun["coverage run<br/>-m pytest"]
    covdata[".coverage<br/>SQLite Database"]
    covreport["coverage report<br/>Terminal Output"]
    covhtml["coverage html<br/>htmlcov/ Directory"]
    
    tests --> covrun
    covrun --> covdata
    covdata --> covreport
    covdata --> covhtml

Coverage Execution

Generate Coverage Data:

# Run tests with coverage
coverage run -m pytest

# Run with branch coverage (recommended)
coverage run --branch -m pytest

# Run with specific source (avoid including tests themselves)
coverage run --source=wshawk -m pytest

View Coverage Reports:

# Terminal report
coverage report

# Terminal report with missing lines
coverage report -m

# Generate HTML report (outputs to htmlcov/)
coverage html

# Open HTML report in browser
open htmlcov/index.html  # macOS
xdg-open htmlcov/index.html  # Linux

Coverage Artifacts

| Artifact | Location | Purpose | Gitignored | |----------|----------|---------|------------| | Coverage data | .coverage | Binary SQLite database with execution traces | Yes | | HTML report | htmlcov/ | Interactive browser-based coverage visualization | Yes | | XML report | coverage.xml | Machine-readable format for CI/CD integration | Yes |

Coverage Thresholds

Critical security modules should maintain high coverage:

• Core Scanner (scanner_v2.py): ≥85% line coverage, ≥75% branch coverage
• Payload System (payloads.py): ≥80% line coverage
• Verification Modules (*_verifier.py): ≥85% line coverage
• Resilience Layer (resilient_session.py): ≥90% line coverage

Sources: .gitignore:40-42

---

Multi-Environment Testing with tox

tox enables testing across multiple Python versions and dependency configurations, ensuring compatibility with the supported Python range (3.8-3.13).

tox Configuration

A typical tox.ini configuration:

[tox]
envlist = py38,py39,py310,py311,py312,py313,lint,coverage

[testenv]
deps =
    pytest
    pytest-asyncio
    pytest-cov
    hypothesis
commands =
    pytest {posargs}

[testenv:coverage]
deps =
    {[testenv]deps}
    coverage
commands =
    coverage run --branch -m pytest
    coverage report -m
    coverage html

[testenv:lint]
deps =
    flake8
    black
    mypy
commands =
    flake8 wshawk/
    black --check wshawk/
    mypy wshawk/

Running tox

# Run all environments
tox

# Run specific environment
tox -e py311

# Run coverage environment
tox -e coverage

# Run lint environment
tox -e lint

# Recreate virtual environments
tox -r

tox Artifacts

The .tox/ directory contains isolated virtual environments for each test configuration. This directory is gitignored to avoid committing large dependency installations.

Sources: setup.py:32-37, .gitignore:42

---

Property-Based Testing with Hypothesis

Hypothesis provides property-based testing, generating hundreds of test cases from high-level specifications to uncover edge cases in payload handling and message parsing.

graph TB
    strategy["@given Strategies<br/>st.text(), st.integers()<br/>Custom Generators"]
    hypothesis["Hypothesis Engine<br/>Example Generation<br/>Shrinking"]
    testfunc["Test Function<br/>test_property_*<br/>Assertions"]
    database[".hypothesis/<br/>Example Database<br/>Failing Cases"]
    
    strategy --> hypothesis
    hypothesis --> testfunc
    testfunc -->|"Failure"| database
    database -->|"Replay"| hypothesis

Hypothesis Test Example

from hypothesis import given, strategies as st
from wshawk.analysis import MessageAnalyzer

class TestMessageAnalyzerProperties:
    @given(st.text(min_size=1, max_size=1000))
    def test_property_analyzer_never_crashes(self, message):
        """MessageAnalyzer should handle any text input without crashing."""
        analyzer = MessageAnalyzer()
        # Should not raise exception
        result = analyzer.analyze_message(message)
        assert result is not None
    
    @given(st.dictionaries(
        keys=st.text(min_size=1, max_size=50),
        values=st.one_of(st.text(), st.integers(), st.floats())
    ))
    def test_property_json_field_detection(self, json_dict):
        """Field detection should work for any valid JSON structure."""
        import json
        analyzer = MessageAnalyzer()
        message = json.dumps(json_dict)
        fields = analyzer.detect_fields(message)
        # Should detect at least the top-level keys
        assert len(fields) >= len(json_dict)

Hypothesis Configuration

Hypothesis stores example databases in .hypothesis/ to replay failing cases:

# tests/conftest.py or specific test file
from hypothesis import settings, HealthCheck

# Custom profile for security testing
settings.register_profile("security", 
    max_examples=1000,  # More examples for thorough testing
    deadline=None,      # No time limit for complex operations
    suppress_health_check=[HealthCheck.too_slow]
)

settings.load_profile("security")

Hypothesis Artifacts

| Artifact | Location | Purpose | Gitignored | |----------|----------|---------|------------| | Example database | .hypothesis/ | Stores generated examples, especially failures | Yes |

Sources: .gitignore:43

---

Test Artifacts and Cleanup

WSHawk generates several test artifacts during execution. The .gitignore configuration ensures these are not committed to version control.

Artifact Locations

graph TB
    subgraph "Test Execution"
        pytest_run["pytest execution"]
        coverage_run["coverage run"]
        tox_run["tox execution"]
        hypothesis_run["hypothesis tests"]
    end
    
    subgraph "Cached Artifacts (Gitignored)"
        pytest_cache[".pytest_cache/<br/>Session cache<br/>Last failed tests"]
        coverage_file[".coverage<br/>Coverage data<br/>SQLite format"]
        htmlcov_dir["htmlcov/<br/>HTML reports<br/>Interactive viewer"]
        tox_dir[".tox/<br/>Virtual environments<br/>Per-Python version"]
        hypothesis_dir[".hypothesis/<br/>Example database<br/>Failing cases"]
    end
    
    subgraph "Test Results (Gitignored)"
        test_results["test_results/<br/>JUnit XML<br/>CI/CD reports"]
        scan_results["scan_results/<br/>Test scan outputs<br/>Sample reports"]
    end
    
    pytest_run --> pytest_cache
    coverage_run --> coverage_file
    coverage_run --> htmlcov_dir
    tox_run --> tox_dir
    hypothesis_run --> hypothesis_dir
    
    pytest_run --> test_results
    pytest_run --> scan_results

Cleanup Commands

# Remove all test artifacts
rm -rf .pytest_cache .coverage htmlcov .tox .hypothesis test_results scan_results

# Remove Python bytecode and build artifacts
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type f -name "*.pyc" -delete
find . -type f -name "*.pyo" -delete
rm -rf build/ dist/ *.egg-info/

# Complete clean (including virtual environments)
rm -rf venv/ env/ .venv/

Sources: .gitignore:38-43, .gitignore:78-82

---

CI/CD Integration

WSHawk's testing infrastructure integrates with GitHub Actions to run tests automatically on every push and pull request.

GitHub Actions Test Workflow

A typical test workflow (.github/workflows/test.yml):

name: Test Suite

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        python-version: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13']
    
    steps:
    - uses: actions/checkout@v4
    
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v4
      with:
        python-version: ${{ matrix.python-version }}
    
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -e .
        pip install pytest pytest-cov hypothesis
    
    - name: Run tests with coverage
      run: |
        pytest --cov=wshawk --cov-report=xml --cov-report=term
    
    - name: Upload coverage to Codecov
      uses: codecov/codecov-action@v3
      with:
        file: ./coverage.xml

Test Report Integration

Test results can be exported in multiple formats for CI/CD integration:

# JUnit XML for Jenkins/GitLab CI
pytest --junitxml=test_results/junit.xml

# Coverage XML for Codecov/Coveralls
pytest --cov=wshawk --cov-report=xml

# HTML report for artifact storage
pytest --html=test_results/report.html --self-contained-html

Sources: setup.py:32-37

---

Writing New Tests

Test Structure Guidelines

Unit Test Template:

# tests/unit/test_new_feature.py
import pytest
from wshawk.new_module import NewClass

class TestNewClass:
    """Tests for NewClass functionality."""
    
    def test_initialization(self):
        """Test that NewClass initializes correctly."""
        instance = NewClass(param="value")
        assert instance.param == "value"
    
    def test_method_success_case(self):
        """Test method behavior in success scenario."""
        instance = NewClass()
        result = instance.method(input="test")
        assert result == "expected"
    
    def test_method_error_case(self):
        """Test method behavior when errors occur."""
        instance = NewClass()
        with pytest.raises(ValueError, match="invalid input"):
            instance.method(input="invalid")
    
    @pytest.mark.parametrize("input,expected", [
        ("test1", "result1"),
        ("test2", "result2"),
        ("test3", "result3"),
    ])
    def test_method_parametrized(self, input, expected):
        """Test method with multiple input cases."""
        instance = NewClass()
        assert instance.method(input=input) == expected

Integration Test Template:

# tests/integration/test_new_workflow.py
import pytest
from wshawk.scanner_v2 import WSHawkV2

@pytest.mark.integration
class TestNewWorkflow:
    """Integration tests for new scanning workflow."""
    
    @pytest.fixture
    def scanner(self):
        """Provide configured scanner instance."""
        scanner = WSHawkV2("ws://localhost:8080")
        scanner.use_headless_browser = False
        return scanner
    
    async def test_full_workflow(self, scanner, mock_vulnerable_server):
        """Test complete workflow from connection to report."""
        # Setup
        await scanner.connect()
        
        # Execute
        results = await scanner.run_heuristic_scan()
        
        # Verify
        assert len(results) > 0
        assert any(v.severity == "HIGH" for v in results)
        
        # Cleanup
        await scanner.disconnect()

Test Data Management

Fixtures for Test Payloads:

# tests/fixtures/sample_payloads.py
import json
import pytest

@pytest.fixture
def xss_payloads():
    """Provides sample XSS payloads for testing."""
    return [
        "<script>alert('xss')</script>",
        "<img src=x onerror=alert(1)>",
        "javascript:alert(document.cookie)",
    ]

@pytest.fixture
def json_message_samples():
    """Provides sample JSON WebSocket messages."""
    return [
        json.dumps({"action": "login", "user": "test"}),
        json.dumps({"type": "message", "content": "hello"}),
        json.dumps({"cmd": "execute", "args": ["ls", "-la"]}),
    ]

Async Test Support

For testing async functionality (WebSocket operations, scanner methods):

import pytest

# Mark test as async
@pytest.mark.asyncio
async def test_async_scan():
    """Test async scanning operations."""
    from wshawk.scanner_v2 import WSHawkV2
    
    scanner = WSHawkV2("ws://test.example.com")
    await scanner.connect()
    result = await scanner.send_payload("test")
    assert result is not None
    await scanner.disconnect()

Install pytest-asyncio dependency:

pip install pytest-asyncio

Sources: setup.py:9-14

---

Test Dependencies

The following dependencies are required for testing but not for production use:

| Package | Version | Purpose | |---------|---------|---------| | pytest | Latest | Test runner and framework | | pytest-cov | Latest | Coverage plugin for pytest | | pytest-asyncio | Latest | Async test support | | pytest-xdist | Latest | Parallel test execution | | hypothesis | Latest | Property-based testing | | coverage | Latest | Code coverage measurement | | tox | Latest | Multi-environment testing |

These are typically installed via a requirements-dev.txt or as extras in pyproject.toml:

[project.optional-dependencies]
dev = [
    "pytest>=7.0",
    "pytest-cov>=4.0",
    "pytest-asyncio>=0.21",
    "pytest-xdist>=3.0",
    "hypothesis>=6.0",
    "coverage>=7.0",
    "tox>=4.0",
]

Install with:

pip install -e ".[dev]"

Sources: setup.py:9-14

---

Summary

WSHawk's testing infrastructure provides comprehensive verification through:

• pytest for flexible test execution with fixtures and parametrization
• tox for multi-Python-version compatibility testing (3.8-3.13)
• coverage.py for measuring code coverage with branch analysis
• hypothesis for property-based testing to uncover edge cases
• CI/CD integration via GitHub Actions for automated testing

Test artifacts (.pytest_cache/, .coverage, htmlcov/, .tox/, .hypothesis/) are gitignored to maintain a clean repository. The test suite is excluded from package distribution via the exclude directive in setup.py.

For more information on development workflows, see 11.1 for environment setup and 11.8 for Docker-based testing.

Sources: setup.py:24, .gitignore:38-43