Comprehensive guide to implementing XARF in your systems for abuse reporting and handling

Implementation Guide

This guide walks you through implementing XARF (eXtended Abuse Reporting Format) in your applications, from basic report generation to advanced integration patterns.

---

Quick Start

1. Choose Your Language

XARF provides libraries for multiple programming languages:

• Python (Beta) - Full-featured library with validation and schema support
• JavaScript/Node.js (Alpha) - Browser and Node.js compatible
• Go (Alpha) - High-performance implementation

2. Install the Library

Python

# alpha — not yet on PyPI
pip install git+https://github.com/xarf/xarf-python.git

JavaScript/Node.js

npm install @xarf/xarf

Go

The Go library is not yet publicly available (coming soon).

3. Create Your First Report

Python Example

from xarf import XARFReport
from datetime import datetime

# Create a new XARF report
report = XARFReport(
    xarf_version="4.0.0",
    report_id="550e8400-e29b-41d4-a716-446655440000",
    timestamp=datetime.utcnow().isoformat() + "Z",
    reporter={
        "org": "Security Operations",
        "contact": "[email protected]",
        "domain": "example.com"
    },
    sender={
        "org": "Security Operations",
        "contact": "[email protected]",
        "domain": "example.com"
    },
    source_identifier="192.0.2.100",
    category="connection",
    type="ddos"
)

# Validate the report
if report.validate():
    print("✓ Report is valid!")

    # Export to JSON
    json_output = report.to_json(indent=2)
    print(json_output)
else:
    print("✗ Validation errors:")
    for error in report.validation_errors:
        print(f"  - {error}")

JavaScript Example

const { XARFReport } = require('xarf');

// Create a new XARF report
const report = new XARFReport({
  xarf_version: '4.0.0',
  report_id: '550e8400-e29b-41d4-a716-446655440000',
  timestamp: new Date().toISOString(),
  reporter: {
    org: 'Security Operations',
    contact: '[email protected]',
    domain: 'example.com'
  },
  sender: {
    org: 'Security Operations',
    contact: '[email protected]',
    domain: 'example.com'
  },
  source_identifier: '192.0.2.100',
  category: 'connection',
  type: 'ddos'
});

// Validate and export
if (report.validate()) {
  console.log('✓ Report is valid!');
  console.log(report.toJSON(null, 2));
} else {
  console.log('✗ Validation errors:');
  report.validationErrors.forEach(error => console.log(`  - ${error}`));
}

---

Validation

Schema Validation

All XARF reports must conform to the JSON Schema for their specific type. The library handles validation automatically:

from xarf import XARFReport, ValidationError

try:
    report = XARFReport.from_json(json_string)
    report.validate(strict=True)  # Raises ValidationError if invalid
except ValidationError as e:
    print(f"Validation failed: {e.message}")
    for error in e.errors:
        print(f"  {error.path}: {error.message}")

Field Requirements

Each XARF type has different field requirements. Use the emoji indicators to identify:

• 🟠 Mandatory - MUST be present in all valid reports
• 🟢 Recommended - SHOULD be included when information is available
• 🔵 Optional - MAY be included for additional context

See Sample Reports for complete field reference for each type.

Online Validator

Use our Schema Validator to validate reports in your browser without installing anything.

---

Integration Patterns

1. Automated Abuse Detection

Use Case: Your system detects malicious activity and automatically generates XARF reports.

class AbuseDetector:
    def __init__(self, xarf_reporter):
        self.xarf_reporter = xarf_reporter

    def on_ddos_detected(self, attack_data):
        """Called when DDoS attack is detected"""
        report = self.xarf_reporter.create_report(
            category="connection",
            abuse_type="ddos",
            source_identifier=attack_data['source_ip'],
            source_port=attack_data['source_port'],
            protocol=attack_data['protocol'],
            evidence=self._collect_evidence(attack_data),
            tags=[
                f"protocol:{attack_data['protocol']}",
                f"volume:{attack_data['gbps']}gbps"
            ]
        )

        # Send to abuse contact
        self.send_report(report, attack_data['source_asn'])

2. Receiving and Processing Reports

Use Case: Your abuse handling system receives XARF reports from external sources.

from xarf import XARFReport

def process_abuse_report(json_data):
    """Process incoming XARF report"""
    try:
        # Parse and validate
        report = XARFReport.from_json(json_data)
        report.validate()

        # Route based on category and type
        if report.category == "connection" and report.abuse_type == "ddos":
            handle_ddos_report(report)
        elif report.category == "vulnerability":
            handle_vulnerability_report(report)

        # Log receipt
        log_report_received(report.report_id, report.reporter.org)

    except ValidationError as e:
        log_invalid_report(json_data, str(e))

3. Batch Processing

Use Case: Process multiple reports efficiently.

from xarf import XARFBatch

def process_daily_reports(report_files):
    """Process all reports from the past 24 hours"""
    batch = XARFBatch()

    # Load all reports
    for file_path in report_files:
        with open(file_path, 'r') as f:
            batch.add_report(f.read())

    # Validate all at once
    results = batch.validate_all()

    # Process valid reports
    for report in results.valid:
        take_action(report)

    # Log invalid reports
    for error in results.invalid:
        log_validation_failure(error.report_id, error.errors)

4. Format Conversion

Use Case: Convert between XARF and other abuse reporting formats.

from xarf.converters import ARFConverter, IODefConverter

# Convert from ARF to XARF
arf_data = load_arf_report('report.xml')
xarf_report = ARFConverter.to_xarf(arf_data)

# Convert XARF to IODEF
iodef_xml = IODefConverter.from_xarf(xarf_report)

---

Error Handling

Common Errors

Missing Mandatory Fields

{
  "error": "ValidationError",
  "message": "Missing required field: 'reporter'",
  "path": "$",
  "expected": "object with 'org', 'contact', 'domain'"
}

Solution: Ensure all mandatory fields (🟠) are present.

Invalid Field Format

{
  "error": "ValidationError",
  "message": "Invalid timestamp format",
  "path": "$.timestamp",
  "expected": "ISO 8601 format (YYYY-MM-DDTHH:mm:ssZ)"
}

Solution: Use proper ISO 8601 format for timestamps.

Unknown Classification or Type

{
  "error": "ValidationError",
  "message": "Unknown type 'unknown_type' for category 'connection'",
  "path": "$.type",
  "expected": "One of: ddos, port-scan, login-attack, brute-force, etc."
}

Solution: Use only valid types for each category. See Content Types.

Best Practices

1. Always validate before sending - Use report.validate() to catch errors early
2. Log validation failures - Keep track of invalid reports for debugging
3. Graceful degradation - Handle missing recommended (🟢) fields gracefully
4. Version compatibility - Check xarf_version field when parsing reports

---

Security & Privacy

Handling Sensitive Data

XARF reports may contain sensitive information. Follow these guidelines:

1. Evidence Protection

from xarf.security import SecureEvidence

# Encrypt evidence before storage
evidence = SecureEvidence.encrypt(
    content=raw_evidence,
    encryption_key=get_encryption_key(),
    algorithm="AES-256-GCM"
)

report.add_evidence(
    content_type="application/octet-stream",
    description="Encrypted packet capture",
    payload=evidence.to_base64()
)

2. PII Redaction

from xarf.privacy import PIIRedactor

# Automatically redact PII from evidence
redactor = PIIRedactor()
clean_evidence = redactor.redact(evidence_text)

# Redaction preserves structure but removes:
# - Email addresses
# - Phone numbers
# - Credit card numbers
# - Personal identifiers

3. Access Control

from xarf.security import AccessControl

# Define who can access reports
acl = AccessControl()
acl.set_reader("[email protected]")
acl.set_reader("[email protected]")

# Embed access control in report
report.metadata['access_control'] = acl.to_dict()

Transport Security

Always use TLS/HTTPS when transmitting XARF reports:

import requests

response = requests.post(
    'https://abuse.example.com/xarf',
    json=report.to_dict(),
    headers={'Content-Type': 'application/json'},
    verify=True  # Always verify SSL certificates
)

---

Testing

Unit Tests

import unittest
from xarf import XARFReport

class TestXARFReports(unittest.TestCase):

    def test_valid_ddos_report(self):
        """Test creating a valid DDoS report"""
        report = XARFReport(
            xarf_version="4.0.0",
            report_id="test-001",
            timestamp="2024-01-15T10:00:00Z",
            reporter={
                "org": "Test Security",
                "contact": "[email protected]",
                "domain": "example.com"
            },
            sender={
                "org": "Test Security",
                "contact": "[email protected]",
                "domain": "example.com"
            },
            source_identifier="192.0.2.100",
            category="connection",
            type="ddos"
        )

        self.assertTrue(report.validate())

    def test_missing_mandatory_field(self):
        """Test that missing mandatory field raises error"""
        with self.assertRaises(ValidationError):
            report = XARFReport(
                xarf_version="4.0.0",
                # Missing report_id
                timestamp="2024-01-15T10:00:00Z",
                reporter={
                    "org": "Test",
                    "contact": "[email protected]",
                    "domain": "example.com"
                },
                sender={
                    "org": "Test",
                    "contact": "[email protected]",
                    "domain": "example.com"
                },
                source_identifier="192.0.2.100",
                category="connection",
                type="ddos"
            )
            report.validate(strict=True)

Integration Tests

def test_end_to_end_workflow():
    """Test complete report generation and submission"""
    # 1. Detect abuse
    attack_data = detect_attack()

    # 2. Generate report
    report = generate_xarf_report(attack_data)

    # 3. Validate
    assert report.validate()

    # 4. Submit
    response = submit_report(report)
    assert response.status_code == 200

    # 5. Verify acknowledgment
    ack = parse_acknowledgment(response.json())
    assert ack.report_id == report.report_id

Test Data

Create test reports using the sample files in the specification or use the libraries to generate valid XARF reports programmatically.

---

Production Checklist

Before deploying XARF in production:

Infrastructure

• TLS/HTTPS configured for all report transmission
• Authentication implemented for report endpoints
• Rate limiting configured to prevent abuse
• Monitoring set up for report volume and errors
• Logging configured for audit trail
• Backup system for reports and evidence

Security

• Encryption enabled for sensitive evidence
• PII redaction implemented where required
• Access control policies defined
• Data retention policy configured
• Security review completed
• Penetration testing performed

Compliance

• GDPR compliance verified (if applicable)
• Data protection regulations reviewed
• Legal review completed
• Privacy policy updated
• Terms of service include XARF handling

Operational

• Documentation completed for operations team
• Runbooks created for common scenarios
• On-call procedures defined
• Escalation paths documented
• Performance testing completed
• Disaster recovery plan in place

Validation

• Schema validation enabled in production
• Error handling tested thoroughly
• Logging captures all validation failures
• Monitoring alerts on validation errors
• Fallback procedures defined

---

Next Steps

• Sample Reports - Explore all event types with examples
• Schema Reference - Detailed schema documentation
• Best Practices - Advanced tips and patterns
• Python Library - Full API documentation
• Tools - Validators, generators, and converters

---

Need Help?

• GitHub Discussions - Ask questions and share experiences
• GitHub Issues - Report bugs or request features
• Contributing - Help improve XARF

---

Ready to implement? Start with our Python Library or explore Sample Reports for your specific use case.