Molecule Testing

The solti-matrix-mgr collection uses Molecule for testing across local development and CI/CD environments.

Test Scenarios Overview

Scenario	Environment	Purpose	Usage
default	Local Matrix	Test matrix_event module	molecule test
user-mgmt	Local Matrix	Test user/room management with user_type	molecule test -s user-mgmt
e2e	GitHub Actions	Module primitives integration	Automatic on push/PR
apply-config	GitHub Actions	Declarative workflow (inventory → converge)	Automatic on push/PR (planned)

Testing Strategy

The collection uses a two-layer testing approach:

Layer 1: Module Primitives (e2e, user-mgmt, default)

Tests individual module functionality with direct calls. Validates that each module works correctly in isolation.

Purpose: Prove the foundation works Pattern: Imperative - call modules directly with explicit parameters

Layer 2: Production Workflow (apply-config - planned)

Tests the declarative inventory → apply-config.yml workflow used in real deployments.

Purpose: Validate production DevOps pipeline

Pattern: Declarative - define desired state in inventory, converge with playbook

Use case: Deploy Matrix infrastructure for CI/CD with public/private room access

Environment: GitHub Actions only (too destructive for local Proxmox server)

Local Testing Scenarios

Prerequisites

All local scenarios require access to a live Matrix homeserver with admin credentials.

Set environment variables:

source ~/.secrets/LabMatrix
# Or export manually:
export MATRIX_HOMESERVER_URL="https://matrix.example.com"
export MATRIX_ADMIN_USER="@admin:example.com"
export MATRIX_ADMIN_PASSWORD="your-password"
export MATRIX_ADMIN_TOKEN="syt_..."

Scenario: default

Purpose: Test matrix_event module functionality

Tests: - Event posting to Matrix rooms - Self-healing authentication - Event content validation

Run:

cd solti-matrix-mgr
molecule test

What it does: 1. Posts test event to configured room 2. Verifies event was received 3. Tests token auto-refresh on expiration 4. Validates event structure

Artifacts: ~/.ansible/tmp/molecule.*/verify_output/matrix_event_result.json

Scenario: user-mgmt

Purpose: Test user and room management with user_type support

Tests: - synapse_user module with user_type: bot - synapse_user_info module (query/filter by type) - synapse_room_info module (query by alias) - Idempotent user creation - Bot account MAU exemption

Run:

cd solti-matrix-mgr
molecule test -s user-mgmt

Test flow: 1. Create 2 bot users with user_type: "bot" 2. Create test room 3. Query bot users via synapse_user_info 4. Verify user_type field is set correctly 5. Query room via synapse_room_info 6. Re-run user creation (idempotency check) 7. Verify no changes on second run

Validation:

- name: Verify both test bots exist
  ansible.builtin.assert:
    that:
      - bot_users.total >= 2
      - bot_users.users | selectattr('name', 'contains', 'test-bot1') | list | length == 1
      - bot_users.users | selectattr('name', 'contains', 'test-bot2') | list | length == 1

- name: Verify bot users have correct user_type
  ansible.builtin.assert:
    that:
      - item.user_type == 'bot'
  loop: "{{ bot_users.users | selectattr('name', 'contains', 'test-bot') | list }}"

CI/CD Testing (GitHub Actions)

Scenario: e2e

Purpose: End-to-end integration test with Docker Synapse in GitHub Actions

Environment: Docker Synapse + Postgres (ephemeral)

Tests: - User lifecycle (create, use, delete with user_type: bot) - Room management - Two-bot interaction (sender/receiver bots on separate "servers") - Deletion and deactivation - Idempotency

Runs automatically on: - Push to main or test branches - Pull requests to main

View results:

gh run list --workflow=e2e.yml
# Or visit GitHub Actions page

Details: See .github/workflows/e2e.yml and extensions/molecule/e2e/

Running Tests Locally

First Time Setup

cd solti-matrix-mgr

# Create Python venv
python3 -m venv venv
source venv/bin/activate

# Install Molecule
pip install molecule molecule-plugins[docker] ansible-core

# Set credentials
source ~/.secrets/LabMatrix

Run All Local Tests

source venv/bin/activate
source ~/.secrets/LabMatrix

# Run default scenario
molecule test

# Run user-mgmt scenario
molecule test -s user-mgmt

Development Workflow

# Create test environment (without cleanup)
molecule create
molecule converge

# Run verification
molecule verify

# Inspect if needed
molecule login

# Clean up when done
molecule destroy

Planned: apply-config Scenario (GitHub Actions)

Purpose

Test the declarative DevOps pipeline used in production deployments. Based on real mylab inventory pattern.

Environment: GitHub Actions only - spins up ephemeral Docker Synapse (too destructive for local Proxmox)

Inventory Pattern (Sanitized from mylab)

# group_vars/matrix_svc/main.yml
matrix_domain: "example.com"
matrix_homeserver_url: "https://matrix.example.com"

# Users - infrastructure as code
matrix_users:
  # Admin account
  - user_id: "@admin"
    displayname: "Admin"
    role: admin
    password: "{{ vault_matrix_admin_password }}"
    state: present

  # Bot accounts for automation
  - user_id: "@deploy-bot"
    displayname: "Deployment Bot"
    role: bot
    password: "{{ vault_matrix_deploy_bot_password }}"
    state: present

  - user_id: "@verify-bot"
    displayname: "Verification Bot"
    role: bot
    password: "{{ vault_matrix_verify_bot_password }}"
    state: present

  # Human users
  - user_id: "@ops-user"
    displayname: "Operations User"
    role: user
    password: "{{ vault_matrix_ops_user_password }}"
    state: present

# Rooms - declarative configuration
matrix_rooms:
  # Public deployment notifications
  - alias: "deploys"
    name: "Deployment Notifications"
    topic: "CI/CD deployment results"
    preset: "public_chat"
    state: present
    members:
      - user_id: "@deploy-bot"
        power_level: 50
        auto_join: true
      - user_id: "@ops-user"
        power_level: 100
        auto_join: true

  # Private verification room
  - alias: "verify"
    name: "Verification Results"
    topic: "Automated test results from Molecule/CI"
    preset: "private_chat"
    state: present
    members:
      - user_id: "@verify-bot"
        power_level: 50
        auto_join: true
      - user_id: "@ops-user"
        power_level: 100
        auto_join: true

  # Encrypted ops room
  - alias: "team"
    name: "Team Operations"
    topic: "Secure team communications"
    preset: "private_chat"
    encrypted: true
    state: present
    members:
      - user_id: "@admin"
        power_level: 100
        auto_join: true
      - user_id: "@ops-user"
        power_level: 100
        auto_join: true

Playbook Flow

# Test workflow
- name: Apply Matrix Configuration
  hosts: localhost
  tasks:
    - name: Load inventory vars
      include_vars: group_vars/matrix_svc/main.yml

    - name: Converge to desired state
      include_tasks: playbooks/matrix/apply-config.yml

    - name: Verify infrastructure
      include_tasks: verify.yml

What It Tests

• ✅ Declarative user provisioning from inventory
• ✅ Declarative room creation (public/private/encrypted)
• ✅ Auto-join bot accounts to rooms
• ✅ Password resolution from environment vars
• ✅ Full DevOps pipeline: inventory → converge → verify
• ✅ Idempotency of apply-config.yml
• ✅ Token generation and caching

Why It Matters

Current tests (e2e): Prove modules work This test: Proves the production workflow works

Real deployments use inventory-driven configuration management, not direct module calls. This scenario validates the entire automation pipeline.

Modules Tested

• ✅ matrix_event - Event posting (default)
• ✅ synapse_user - User creation with user_type (user-mgmt, e2e)
• ✅ synapse_user_info - User queries by type (user-mgmt, e2e)
• ✅ synapse_room - Room management (e2e)
• ✅ synapse_room_info - Room queries (user-mgmt)
• ✅ synapse_device_info - Token auditing (planned)
• ⚠️ apply-config.yml workflow - Not tested in Molecule (gap)

Features Tested

• ✅ Self-healing authentication (auto token refresh)
• ✅ User type classification (bot, support)
• ✅ MAU exemption for bot accounts
• ✅ Idempotent operations
• ✅ User lifecycle (create → use → delete)
• ✅ Room management
• ✅ Two-bot communication
• ✅ Deletion and deactivation
• ✅ Query and filtering

Platforms Tested

• Local: Live Matrix homeserver (Synapse)
• CI/CD: Docker Synapse (ephemeral, isolated)

Troubleshooting Tests

Test Fails: Missing Credentials

TASK [matrix_event] ************************************************************
fatal: [instance]: FAILED! => {"msg": "Missing required parameters: homeserver_url"}

Fix:

source ~/.secrets/LabMatrix
molecule test

Test Fails: Connection Refused

Failed to connect to https://matrix.example.com

Check:

• Homeserver URL is correct
• Network connectivity
• Homeserver is running
• Firewall rules

Test Fails: 403 Forbidden

HTTP 403: User @admin:example.com is not a server admin

Fix: Ensure admin user has admin privileges in Synapse config

E2E Test Fails in CI

Check GitHub Actions logs:

gh run view <run-id> --log

Common issues:

• Docker service not started
• Synapse startup timeout
• Port conflicts

Adding New Tests

Local Scenario

1. Create directory: extensions/molecule/my-scenario/
2. Add molecule.yml, converge.yml, verify.yml
3. Update extensions/molecule/README.md
4. Update this testing guide

CI/CD Integration

1. Add scenario to .github/workflows/e2e.yml
2. Configure Docker services if needed
3. Set environment variables
4. Add status badge to README

External Resources

• Molecule Documentation
• GitHub Actions
• Synapse Docker

Next Steps

• Module Reference - See module source
• Playbook Examples - Common usage patterns
• Token Management - Credential handling