# Enterprise Audit Logging System

This document describes the centralized audit implementation added to this codebase.

## 1) ClickHouse DDL

Reference DDL file:

- `database/clickhouse/audit_logs.sql`

Runtime creation is also available via:

- `php artisan audit:clickhouse-setup`

## 2) Kafka Producer Design

Recommended producer shape (for high-volume clusters):

- Topic: `audit.logs.v1`
- Partition key: `app_id:user_id` (preserves user-level order)
- Value payload: same JSON envelope as `ProcessAuditLogJob`
- Acks: `all`
- Compression: `lz4` or `zstd`
- Retry policy: exponential backoff with dead-letter topic (`audit.logs.dlq`)

Current implementation uses Laravel queue transport (`AUDIT_QUEUE_CONNECTION`) and can be swapped to a Kafka-backed queue worker without changing payload contract.

## 3) Kafka Consumer Design

Consumer responsibilities:

- Parse incoming audit envelope
- Validate required fields (`event_time`, `action_type`, `module_name`, `app_id`)
- Batch by size/time window (e.g., 500 rows or 1s)
- Insert into ClickHouse using `INSERT ... FORMAT JSONEachRow`
- Push malformed payloads to DLQ

Current code path:

- Job: `App\Jobs\ProcessAuditLogJob`
- Writer: `App\Services\Audit\AuditClickhouseService`

## 4) API Middleware Design

Request capture middleware:

- `App\Http\Middleware\CaptureAuditEvent`
- Registered in `bootstrap/app.php` (web middleware append)
- Feature flag: `AUDIT_LOG_ENABLED`
- Skip rules: `AUDIT_IGNORED_PATHS`
- Captures request/response metadata and publishes async job

Auth events are captured centrally via:

- `App\Services\Logs\AuthEventLogger` -> `AuditEventFactory` -> `AuditEventPublisher`

## 5) Laravel/PHP Implementation Example

Core components:

- Config: `config/audit.php`
- Event factory: `App\Services\Audit\AuditEventFactory`
- Action resolver: `App\Services\Audit\AuditActionResolver`
- Diffing: `App\Services\Audit\AuditDiffService`
- Async publisher: `App\Services\Audit\AuditEventPublisher`
- Queue consumer: `App\Jobs\ProcessAuditLogJob`
- ClickHouse writer: `App\Services\Audit\AuditClickhouseService`
- Dashboard queries: `App\Services\Audit\AuditDashboardService`
- APIs: `App\Http\Controllers\Audit\AuditLogController`

## 6) Dashboard Query Examples

Implemented API endpoints:

- `GET /admin/api/audit/summary`
- `GET /admin/api/audit/charts`
- `GET /admin/api/audit/details`
- `GET /admin/api/audit/filters`

Supported filters:

- `date_from`, `date_to`, `user_id`, `module_name`, `action_type`, `entity_id`, `app_id`, `ip_address`, `status`, `search`

## 7) Partitioning and Indexing Strategy

Implemented strategy:

- `PARTITION BY toYYYYMM(event_time)` for bounded monthly reads/writes
- `ORDER BY (app_id, module_name, action_type, user_id, event_time)` for common access patterns
- `LowCardinality` for dimensions (`module_name`, `action_type`, `request_type`, etc.)
- `Array(String)` for `changed_fields`

## 8) Data Retention Strategy

Retention controls:

- `AUDIT_RETENTION_DAYS` (used in runtime table creation)
- Optional ClickHouse TTL clause in runtime setup: `TTL event_time + toIntervalDay(N)`

Recommended:

- Hot data: 90-180 days in primary cluster
- Cold data: periodic export to object storage

## 9) High Volume Scaling Strategy

Recommended production strategy:

- Queue buffering with horizontal workers
- Kafka partitioning by tenant/user
- Batch inserts into ClickHouse (500-5000 rows per insert)
- Dedicated audit cluster or database for isolation
- Materialized views for dashboard rollups
- Query guardrails (date-range defaults, pagination)

## 10) Security and Compliance Best Practices

Implemented/available controls:

- Sensitive field redaction: `AUDIT_REDACT_FIELDS`
- Feature flag enablement: `AUDIT_LOG_ENABLED`
- Guest capture toggle: `AUDIT_CAPTURE_GUESTS`
- Route skipping for sensitive/system paths: `AUDIT_IGNORED_PATHS`

Recommended additional controls:

- Encrypt transport to ClickHouse (`https`)
- Restrict ClickHouse user privileges to append/query-only scope
- Mask PII when exporting audit data
- Monitor failed ingestion and queue backlog with alerts
- Enforce least privilege for dashboard APIs