The 18-Signal
Evaluation Framework
A dual-axis epistemological model for evaluating information quality. Source Credibility and Claim Validity, measured through 18 discrete signals across three categories.
Two Dimensions of Evaluation
Every evaluation measures two independent dimensions. A credible source can make a weak claim. A novel source can produce rigorous methodology. Both axes must be evaluated independently.
Source Credibility
Who produced this information? What is their track record? Are their funding sources transparent? Do they correct errors? Source credibility is evaluated through 6 signals that measure the reliability and transparency of the information's origin — independent of whether any specific claim is true.
Claim Validity
What does the evidence actually support? Is the methodology sound? Are conclusions proportional to the data? Claim validity is evaluated through 12 signals — 6 methodology signals and 6 claim signals — that measure how well the evidence supports what is being asserted, independent of who asserts it.
Source Signals
6 signals evaluating the origin and track record of the information source.
Publication History
Track record of published work. Frequency, consistency, and domain concentration. A source with decades of consistent output in a single domain scores differently than a new entrant across many fields.
Funding Transparency
Disclosure of financial support, grants, sponsors, and commercial relationships. Full transparency does not guarantee independence, but opacity guarantees suspicion.
Conflict of Interest
Identified relationships between the source and entities that benefit from specific conclusions. Financial, institutional, political, and personal conflicts are evaluated.
Peer Review Status
Whether the work has undergone independent peer review, and the quality of that review process. Pre-print, single-blind, double-blind, and open peer review are scored differently.
Correction History
How the source handles errors. Sources that publish corrections and retractions transparently score higher than those that silently edit or never acknowledge mistakes.
Institutional Affiliation
The institutional context of the source. University research labs, government agencies, corporate R&D, think tanks, and independent researchers each carry different credibility profiles.
Methodology Signals
6 signals evaluating the rigor and transparency of the research process.
Sample Size Adequacy
Whether the sample size is sufficient to support the statistical claims made. Power analysis, effect size, and population representativeness are evaluated.
Control Groups
Presence and appropriateness of control conditions. Randomization, blinding, and matching strategies. Not all research requires control groups — the scoring adapts to research design.
Statistical Rigor
Appropriate use of statistical methods. Pre-registration, multiple comparison corrections, effect sizes alongside p-values, Bayesian alternatives where appropriate.
Reproducibility
Can the work be independently reproduced? Code availability, data access, detailed protocol descriptions, and documented replication attempts.
Data Availability
Access to underlying data. Open data, restricted access with justification, or no access. Data format, documentation, and metadata quality.
Methodology Transparency
Completeness of methodological description. Could another researcher replicate the approach from the description alone? Pre-registration of hypotheses and analysis plans.
Claim Signals
6 signals evaluating the relationship between evidence and assertions.
Internal Consistency
Do the claims within the work contradict each other? Are conclusions aligned with the reported data? Logical coherence across the full document.
External Consistency
Alignment with established findings in the field. Novel claims that contradict established knowledge are not penalized — but they require stronger evidence and explicit acknowledgment of divergence.
Scope vs Evidence
Proportionality of claims to supporting evidence. Broad claims from narrow data score poorly. Appropriately scoped claims from robust data score well.
Confidence Calibration
Does the source express appropriate levels of certainty? Overconfidence in weak evidence and underconfidence in strong evidence are both penalized.
Negative Result Handling
How the work treats null results, failed hypotheses, and unfavorable findings. Negative results are results. Suppression of inconvenient data is a critical integrity failure.
Uncertainty Quantification
Explicit quantification of uncertainty. Confidence intervals, error bars, sensitivity analyses, limitations sections. How well does the source communicate what it does not know?
How Scores Are Calculated
Each signal is scored independently on a 0 to 1 scale. Signals are weighted by category, then aggregated to produce axis scores and an overall epistemological score.
Epistemological Score
and ws, wm, wc are category weights (default equal weighting: 0.33 each)
We rate methodology, not conclusions
A paper with rigorous methodology that reaches an uncomfortable conclusion scores higher than a paper with poor methodology that confirms expectations.
Negative results are results
We do not suppress. We contextualize. Null findings, failed replications, and inconvenient data contribute positively to epistemological scores.
We are the audience, not the expert
Alitheion evaluates the quality of the process, not the domain-specific truth of conclusions. We are methodological auditors, not subject matter authorities.
Transparency over certainty
A source that clearly communicates what it does not know scores higher than one that projects false certainty.