Skip to content

Limitations — What Bukti Does Not Verify

This page documents what Bukti cannot verify, where the system's claims are weaker than they might appear, and which gaps are documented known limitations rather than undetected bugs.

Bukti's credibility depends on honest disclosure of its limits. An overconfident system whose limits are unknown is more dangerous than an honest system with documented gaps. This page is the honest version.

Calibration

All scoring parameters today carry calibrated: false. The evidence weights, half-lives, and prior parameters are theory-informed defaults derived from the predictive-validity literature (primarily Sackett 2022, Arthur 1998, Wang 2013). They are not fitted to Bukti's own outcome data and have not been validated against observed real-world capability assessments.

What this means in practice: the system's predicted substantive scores (e.g., a median of 0.72) are best interpreted as "this entity has a reasonable body of evidence for this capability" — not as "there is a 72% probability this entity is competent." The latter interpretation requires calibration. See calibration-status.md.

Identity verification gaps

Institutional VC signature verification is partial today. When an entity imports an Open Badges 3.0 credential and claims I3 identity grade via that credential, Bukti accepts the credential on database record integrity and logs a warning. The system does not yet fully verify the cryptographic signature against the issuer's DID document or check Rekor log entries. This means I3 via institutional VC is currently a softer guarantee than the I3 definition implies. Closing this gap is on the near-term roadmap.

No commit-signing verification yet. behavioral_artifact VOIs sourced from code commits receive the default weight regardless of whether the commits are cryptographically signed. Sigstore/GPG commit-signing verification is on the near-term roadmap. Until then, unsigned commits and signed commits receive the same treatment.

I4 is defined but has no issuance pipeline. No government-ID or KYC credential verification pathway exists today. I4 grade is reserved for future regulated-profession use cases.

Contradiction detection gaps

Two important contradiction-detection capabilities are not yet wired:

  • LLM-based factual contradiction scanning for complex semantic inconsistencies in evidence text (for example, an entity claiming a product reached many users when the supporting evidence shows it was a small classroom project) is on the roadmap but not active today.
  • Capability-ceiling violation detection — cross-checking claimed competence levels against contemporaneous outcome VOIs — requires structured outcome metadata that the system does not yet produce in all sources.

The active rule layer focuses on the strongest signals (revocation via W3C Bitstring Status List, narrow same-day platform collisions for temporal impossibility). See contradiction-detection.md.

Cohort grouping limitations

Capability-context cohort detection is optional today: in scoring paths where the parent-capability lookup is not provided, closely-related skills from the same source may be treated as fully independent evidence. This is a conservative miss in the direction of over-crediting, not a gaming vector.

Ontology coverage

The seed ontology contains approximately 200 nodes prioritized for the first pilot cohort's domains. Capabilities not covered by the seed that fall below the auto-map embedding-similarity threshold are assigned growth-tier IDs. Growth nodes receive default cluster weights and half-life, which may not accurately reflect the actual decay characteristics of an emerging skill.

What the tier labels do and do not mean

"Verified" does not mean independently confirmed by a human. It means: the entity has I3+ identity binding (at minimum two independent OIDC accounts, or one institutional VC) and the substantive evidence score clears the high-substantive thresholds. No human has reviewed the evidence. No background check has been performed. No interview has been conducted.

"Attested" does not mean independently confirmed. It means: the entity has I2+ identity binding and the substantive evidence score clears the mid-substantive thresholds. The evidence may consist entirely of self-uploaded materials whose authenticity Bukti has not independently confirmed.

"Self-declared" includes high-quality evidence submitted by entities with weak identity binding. An entity with a rich, well-sourced portfolio submitted under an I0 or I1 account receives Self-declared because Bukti cannot connect the evidence to a verifiable person.

What Bukti does not do

  • Background checks
  • Employment verification
  • Academic transcript verification (beyond what an OB 3.0 credential from a trusted issuer provides)
  • Interview assessment
  • Reference verification
  • Criminal record checks
  • Identity verification beyond OIDC and institutional VC binding

Bukti is a capability evidence aggregation and scoring system, not a background screening service. See regulatory.md for the regulatory implications.