WORLDSAR Documentation

6-7. Monitoring, Logs, and Cleanup

Operate runs safely by monitoring status, reading logs correctly, and cleaning artifacts intentionally.

6. Monitoring and Logs

Monitoring commands are mode-aware. In the current Makefile, the default mode is often hpc, so set WORLDSAR_MODE=vm explicitly when you want local-only behavior.

Check job status:

make status
WORLDSAR_MODE=vm make status

In HPC mode, make status runs qstat -u $(PBS_USER). In VM mode, it prints guidance to inspect local logs and process exit status instead of querying PBS.

6.1 Log Inspection

List recent logs generated by the current mode:

make logs
WORLDSAR_MODE=vm make logs

Tail local logs directly while a run is active:

tail -f logs/*.stdout.log
tail -f logs/*.stderr.log

VM runs produce timestamped files in LOG_DIR (for example 20260226_120101_PRODUCT.stdout.log and .stderr.log). HPC runs may also produce scheduler outputs (.o*, .e*) in the same log directory.

6.2 Failure Triage Checklist

When a run fails, verify in this order:

  1. Mode is correct for your environment (vm vs hpc).
  2. PRODUCT points to an existing SAFE directory (name or explicit path).
  3. Runtime assets exist (SIF_IMAGE and .snap userdir).
  4. For HPC mode, qsub and queue/select resources are available.
  5. Use newest stdout/stderr logs for root-cause details before re-running.

7. Useful Cleanup

Core cleanup and maintenance commands:

make clean
make clean-logs
make clean-snap-artifacts
make show-cache
make clean-hf-cache

make clean removes mode-specific output directories (OUTPUT_DIR, TILES_DIR, DB_DIR). make clean-logs removes runtime and scheduler log files in LOG_DIR. make clean-snap-artifacts removes temporary SNAP pull artifacts under SNAP_TMP_DIR. make clean-hf-cache prunes Hugging Face cache data.

7.1 Safe Cleanup by Mode

Use explicit mode flags to avoid cleaning paths from the wrong environment:

WORLDSAR_MODE=vm make clean clean-logs
WORLDSAR_MODE=hpc make clean clean-logs

This is important when default mode is hpc but you are testing locally.

Cleanup Strategy

Use clean before fresh processing when previous outputs may interfere.

Use clean-logs to reduce noise without removing products.

Use clean-snap-artifacts after bootstrap when temp files are no longer needed.

Use clean-hf-cache when reclaiming storage or fixing cache issues.