AceLM Assessment Catalog
Overview
AceLM is designed to provide a comprehensive suite of 31 specialized engineering lenses across six core disciplines. These lenses evaluate repository health, discover correctness and security issues, verify non-functional requirements, and generate prioritized, actionable remediation plans.
The planned assessment suite operates both as an on-demand diagnostic tool (alm assess <concern> and alm assess-all) and as the automated qualification engine during the gated release lifecycle.
Complete 31-Lens Catalog
| Concern | Discipline | Review Focus & Scope |
|---|---|---|
accessibility |
UX & Documentation | WCAG 2.1 AA for web surfaces; keyboard navigation, contrast, screen reader compatibility, and terminal accessibility rubrics. |
api-design |
Architecture & Design | Intentional public contracts across APIs, CLI, events, and data formats; consistency, errors, schemas, backward compatibility, and ergonomics. |
architecture |
Architecture & Design | Cohesion, coupling, subsystem boundaries, extensibility, state flow, configuration discipline, and anti-over/under-engineering. |
bugs |
Correctness & Reliability | Reachable correctness defects, invalid state transitions, data loss/corruption, contract violations, and reproducible incorrect behavior. |
compatibility |
Compatibility & Compliance | Supported platforms and runtimes, backward compatibility, third-party integrations, configuration migration, and upgrade paths. |
compliance |
Compatibility & Compliance | Assessment of applicable regulatory and governance regimes, separating repository evidence from external organizational controls. |
compliance-readiness |
Compatibility & Compliance | Repository-visible readiness for selected regimes (e.g., FIPS, NIST 800-171, CMMC-L2); surfaces concrete evidence gaps without claiming formal certification. |
data-exfiltration |
Security & Privacy | Sensitive-data egress through exports, logs, telemetry, APIs, SDKs, and model calls; egress filtering and leak prevention. |
data-modeling |
Architecture & Design | Canonical schemas, entity identity, lifecycle boundaries, schema evolution, provenance, and historical data integrity. |
documentation |
UX & Documentation | Accurate and complete READMEs, architecture guides, API references, configuration guides, operational manuals, and contributor docs. |
edge-cases |
Correctness & Reliability | Boundary inputs, resource exhaustion, timing and ordering anomalies, intermittent failures, concurrency races, and crash recovery paths. |
functionality |
Correctness & Reliability | Verification that implemented capabilities satisfy specification criteria; identifies missing essential flows and claims-vs-reality gaps. |
generalization |
Architecture & Design | Multi-tenant and multi-environment productization, justified reuse, configurable policy, provider abstraction, and anti-overgeneralization. |
guiding-principles |
Architecture & Design | Verifies implementation adherence to stated architectural and product principles, recording evidence of conformance or divergence. |
intrusion-detection |
Security & Privacy | Security-event signals, audit telemetry, tamper evidence, and actionable detection readiness within repository scope. |
local-leaks |
Security & Privacy | Machine and developer identifiers, private paths, internal hostnames, usernames, and session tokens across code and build artifacts. |
logging-audit |
Security & Privacy | Structured events, audit completeness and tamper-resistance, correlation IDs, privacy-safe log redaction, and debugging observability. |
memory-resources |
Performance & Resources | Memory leaks, unbounded cache growth, file descriptor cleanup, object lifetimes, thread safety, and long-running daemon stability. |
performance |
Performance & Resources | Latency, throughput, hot execution paths, algorithmic complexity, database queries, memory allocations, cold start, and horizontal scaling. |
privacy |
Security & Privacy | Personal data collection boundaries, purpose limitation, third-party sharing, retention and deletion policies, and data-subject request handling. |
prose |
UX & Documentation | Clarity, precision, tone, and readability across documentation, UI copy, error messages, and release notes. |
ransomware-resilience |
Security & Privacy | Backup and restore verification, storage immutability, destructive-write blast radius limits, data integrity, and disaster recovery readiness. |
reliability |
Correctness & Reliability | Failure isolation, bounded retries, idempotency, graceful degradation, circuit breakers, high availability, and safe resume. |
secrets |
Security & Privacy | Deterministic, redacted scanning of working trees and Git history for credentials, private keys, and API tokens; rotate-first remediation. |
security |
Security & Privacy | Authentication, 2FA/MFA enforcement, RBAC/ABAC authorization, tenant isolation, input sanitization, transport encryption, and dependency hardening. |
self-documentation |
UX & Documentation | Developer discoverability, meaningful naming conventions, contextual help, onboarding ergonomics, and actionable CLI error messages. |
supply-chain |
Security & Privacy | Direct and transitive dependency vulnerabilities, license compatibility, maintainer reputation, dependency pinning, and build provenance. |
testing |
Correctness & Reliability | Meaningful regression suites, integration tests, contract verification, failure simulation, and performance/accessibility test coverage backed by logs. |
threat-model |
Security & Privacy | Trust boundaries, attack surfaces, threat vectors, defense-in-depth posture, and security hardening gaps. |
ui-ux |
UX & Documentation | User journeys, navigation structure, interaction consistency, state feedback, error recovery, and expert efficiency. |
use-cases |
Correctness & Reliability | Primary and alternative actor scenarios, onboarding/offboarding workflows, edge workflows, and end-to-end journey validation. |
Invocation and Operating Modes
AceLM exposes the assessment catalog through unified CLI commands and interactive workspace controls:
bash
# Assess a single engineering concern
alm assess <concern> [--scope <path>]
# Run a consolidated multi-concern assessment
alm assess-all [--category <name>]Supported aliases include a11y (accessibility), perf (performance), sec (security), docs (documentation), and deps (supply-chain).
- Single-Concern Assessment: Produces structured findings, empirical evidence, and targeted remediation proposals.
- Cross-Concern Assessment (
assess-all): Evaluates selected categories or the full catalog, de-duplicates related findings, cross-prioritizes recommendations, and synthesizes a unified remediation plan.
Automated Release Qualification
During the automated release lifecycle, assessments are executed sequentially rather than in an uncoordinated batch:
- Per-Lens Inner Loop: For each lens in the configured release profile, AceLM assesses the candidate, generates required remediation plans, conducts independent adversarial review and human acceptance, executes changes in isolated worktrees, and verifies the result.
- Multi-Pass Convergence:
- Pass 1 (Deep): Comprehensive discovery and remediation across all configured lenses (e.g., 31 evaluations).
- Pass 2 (Light): Verification that subsequent lens changes did not introduce regressions or invalidate previous evidence (31 evaluations).
- Pass 3 (Convergence, if required): Entire configured lens sequence with narrower instructions focused on remaining significant findings and newly affected surfaces before release authorization.
Discipline Guarantees
- Empirical Evidence: Runnable tests, scan logs, and deterministic verification take precedence over agent self-assertions.
- Separation of Concerns: Authoring, reviewing, and verification tasks are performed in isolated agent contexts to prevent confirmation bias.
- Human Authority: Assessment results inform release gates; acceptance and release authority remain with authorized humans.