# Data Freshness Monitor Runbook

## Purpose
`monitor:data-freshness` checks latest timestamps across configured PostgreSQL and ClickHouse sources and sends alert email when any source is stale.

Configured schedule in code:
- `routes/console.php`: `0 * * * *` (hourly, at minute `00`)

## Command Reference
- Dry run (no email send):  
`php artisan monitor:data-freshness --dry-run`
- Live run (email can be sent):  
`php artisan monitor:data-freshness`
- List schedule:  
`php artisan schedule:list`

## Required Scheduler Setup
Laravel scheduled commands run only if the scheduler is running.

Use one of these:

1. System cron (recommended)
- Add cron entry (example):
`* * * * * cd /var/www/php84/sms_admin_es && php artisan schedule:run >> /dev/null 2>&1`

2. Long-running worker
- Run:
`php artisan schedule:work`
- Keep it alive using systemd/supervisor.

If neither is configured, `monitor:data-freshness` will never run automatically.

## Environment Variables
Main keys:
- `DATA_FRESHNESS_MONITOR_ENABLED=true`
- `DATA_FRESHNESS_THRESHOLD_HOURS=6`
- `DATA_FRESHNESS_ALERT_COOLDOWN_SECONDS=3600`
- `DATA_FRESHNESS_ALERT_EMAILS=...`
- `DATA_FRESHNESS_CLICKHOUSE_DATABASE=clickhouse`
- `DATA_FRESHNESS_CLICKHOUSE_TABLE=razorpay_webhooks`
- `DATA_FRESHNESS_CLICKHOUSE_TIMESTAMP_COLUMN=received_at`

## Interpreting Command Output
Result `reason` values:
- `healthy`: all checks are fresh
- `dry_run`: stale/check failures found, but email intentionally skipped
- `cooldown_active`: alert suppressed due to cooldown
- `no_recipients`: no valid emails in `DATA_FRESHNESS_ALERT_EMAILS`
- `send_failed`: email send failed (SMTP/network/auth)
- `sent`: alert mail sent
- `disabled`: monitor disabled via env

Useful debug fields:
- `Reason detail`
- `Cooldown key`
- `Cooldown seconds`
- `Mail error` (when present)
- per-source `Why:` lines

## Why Email May Not Arrive
1. Scheduler not running
- Command is scheduled but never executed.

2. Cooldown active
- Same stale condition already alerted recently.
- Default cooldown is `3600` seconds.

3. Mail transport failure
- SMTP DNS/connectivity/auth issue.
- Check output `Mail error` and application logs.

4. Recipients not configured/invalid
- Validate `DATA_FRESHNESS_ALERT_EMAILS`.

## Log Locations
- Scheduled command output target:  
`storage/logs/data_freshness_monitor.log`
- Application logs:  
`storage/logs/laravel.log` (or stderr, based on logging channel stack)

## Quick Verification Checklist
1. `php artisan schedule:list` shows `monitor:data-freshness`.
2. Scheduler is running (`cron` with `schedule:run` or `schedule:work` process).
3. `php artisan monitor:data-freshness --dry-run` works and shows source status.
4. `php artisan monitor:data-freshness` returns either `sent` or clear reason (`cooldown_active` / `send_failed`).
5. If needed, send direct test mail using app mail config to validate SMTP independently.