Data Analysis
quality-assessment
Automates critical appraisal and quality assessment for research papers by analyzing text against established methodological standards (such as risk of bias tools, quality checklists, or reporting guidelines) and synthesizing a structured evaluation report. Use when you need to assess the methodological quality, internal validity, or reporting completeness of any type of study—including RCTs, observational studies, systematic reviews, qualitative research, or diagnostic accuracy studies.
87100Total Score
Core Capability
88 / 100
Functional Suitability
11 / 12
Reliability
10 / 12
Performance & Context
8 / 8
Agent Usability
14 / 16
Human Usability
8 / 8
Security
10 / 12
Maintainability
10 / 12
Agent-Specific
17 / 20
Medical Task
15 / 20 Passed
87Automates critical appraisal and quality assessment for research papers by analyzing text against established methodological standards (such as risk of bias tools, quality checklists, or reporting guidelines) and synthesizing a structured evaluation report
3/4
86Automates critical appraisal and quality assessment for research papers by analyzing text against established methodological standards (such as risk of bias tools, quality checklists, or reporting guidelines) and synthesizing a structured evaluation report
3/4
86Automates critical appraisal and quality assessment for research papers by analyzing text against established methodological standards (such as risk of bias tools, quality checklists, or reporting guidelines) and synthesizing a structured evaluation report
3/4
86Packaged executable path(s): scripts/extract_pdf.py
3/4
86End-to-end case for Scope-focused workflow aligned to: Automates critical appraisal and quality assessment for research papers by analyzing text against established methodological standards (such as risk of bias tools, quality checklists, or reporting guidelines) and synthesizing a structured evaluation report. Use when you need to assess the methodological quality, internal validity, or reporting completeness of any type of study—including RCTs, observational studies, systematic reviews, qualitative research, or diagnostic accuracy studies
3/4
Veto GatesRequired pass for any deployment consideration
Skill Veto✓ All 4 gates passed
✓
Operational Stability
System remains stable across varied inputs and edge cases
PASS✓
Structural Consistency
Output structure conforms to expected skill contract format
PASS✓
Result Determinism
Equivalent inputs produce semantically equivalent outputs
PASS✓
System Security
No prompt injection, data leakage, or unsafe tool use detected
PASSResearch Veto✅ PASS — Applicable
| Dimension | Result | Detail |
|---|---|---|
| Scientific Integrity | PASS | Scientific integrity held because extraction and analysis outputs stayed tied to provided text, metadata, or runtime evidence rather than invented study findings. |
| Practice Boundaries | PASS | The evaluated outputs stayed inside the Automates critical appraisal and quality assessment for research papers by analyzing text... and did not drift into unsupported interpretation beyond the available inputs. |
| Methodological Ground | PASS | The workflow stayed grounded in its declared rubric or scale-selection logic rather than improvised criteria. |
| Code Usability | PASS | No code-usability failure was preserved for quality-assessment in the legacy evaluation. |
Core Capability88 / 100 — 8 Categories
Functional Suitability
Functional suitability was softened by the legacy issue 'Improve stress-case output rigor'. Stress and boundary scenarios show weaker consistency
11 / 12
92%
Reliability
The archived deduction in reliability traces back to: Improve stress-case output rigor. Stress and boundary scenarios show weaker consistency
10 / 12
83%
Performance & Context
The legacy audit gave full marks to performance context for this package.
8 / 8
100%
Agent Usability
The archived review left some headroom in how quickly an agent can lock onto the intended analysis path.
14 / 16
88%
Human Usability
No point loss was recorded for human usability in the legacy audit.
8 / 8
100%
Security
The packaged workflow stayed safe overall, with only a small remaining deduction around boundary signaling.
10 / 12
83%
Maintainability
The analysis package is maintainable overall, though the archived score suggests modest cleanup headroom.
10 / 12
83%
Agent-Specific
The archived deduction in agent specific traces back to: Stabilize executable path and fallback behavior. Some inputs only reached PARTIAL due to execution gaps or weak boundary handling
17 / 20
85%
Core Capability Total88 / 100
Medical TaskExecution Average: 86.2 / 100 — Assertions: 15/20 Passed
87
Canonical
Automates critical appraisal and quality assessment for research papers by analyzing text against established methodological standards (such as risk of bias tools, quality checklists, or reporting guidelines) and synthesizing a structured evaluation report
3/4 ✓
86
Variant A
Automates critical appraisal and quality assessment for research papers by analyzing text against established methodological standards (such as risk of bias tools, quality checklists, or reporting guidelines) and synthesizing a structured evaluation report
3/4 ✓
86
Edge
Automates critical appraisal and quality assessment for research papers by analyzing text against established methodological standards (such as risk of bias tools, quality checklists, or reporting guidelines) and synthesizing a structured evaluation report
3/4 ✓
86
Variant B
Packaged executable path(s): scripts/extract_pdf.py
3/4 ✓
86
Stress
End-to-end case for Scope-focused workflow aligned to: Automates critical appraisal and quality assessment for research papers by analyzing text against established methodological standards (such as risk of bias tools, quality checklists, or reporting guidelines) and synthesizing a structured evaluation report. Use when you need to assess the methodological quality, internal validity, or reporting completeness of any type of study—including RCTs, observational studies, systematic reviews, qualitative research, or diagnostic accuracy studies
3/4 ✓
87
Canonical✅ Pass
Automates critical appraisal and quality assessment for research papers by analyzing text against established methodological standards (such as risk of bias tools, quality checklists, or reporting guidelines) and synthesizing a structured evaluation report
This canonical case was mostly intact, but the archived review centered its concern on: The script execution path completed successfully for the documented case.
Basic 33/40|Specialized 54/60|Total 87/100
✅A1The quality-assessment output structure matches the documented deliverable
❌A2The script execution path completed successfully for the documented case
✅A3The output stays fully within the documented skill boundary
✅A4The response quality is acceptable for the documented path
Pass rate: 3 / 4
86
Variant A✅ Pass
Automates critical appraisal and quality assessment for research papers by analyzing text against established methodological standards (such as risk of bias tools, quality checklists, or reporting guidelines) and synthesizing a structured evaluation report
The main issue in this variant a run was: The script execution path completed successfully for the documented case.
Basic 31/40|Specialized 55/60|Total 86/100
✅A1The quality-assessment output structure matches the documented deliverable
❌A2The script execution path completed successfully for the documented case
✅A3The output stays fully within the documented skill boundary
✅A4The response quality is acceptable for the documented path
Pass rate: 3 / 4
86
Edge✅ Pass
Automates critical appraisal and quality assessment for research papers by analyzing text against established methodological standards (such as risk of bias tools, quality checklists, or reporting guidelines) and synthesizing a structured evaluation report
The main issue in this edge run was: The script execution path completed successfully for the documented case.
Basic 30/40|Specialized 56/60|Total 86/100
✅A1The quality-assessment output structure matches the documented deliverable
❌A2The script execution path completed successfully for the documented case
✅A3The output stays fully within the documented skill boundary
✅A4The response quality is acceptable for the documented path
Pass rate: 3 / 4
86
Variant B✅ Pass
Packaged executable path(s): scripts/extract_pdf.py
This variant b case was mostly intact, but the archived review centered its concern on: The script execution path completed successfully for the documented case.
Basic 29/40|Specialized 57/60|Total 86/100
✅A1The quality-assessment output structure matches the documented deliverable
❌A2The script execution path completed successfully for the documented case
✅A3The output stays fully within the documented skill boundary
✅A4The response quality is acceptable for the documented path
Pass rate: 3 / 4
86
Stress✅ Pass
End-to-end case for Scope-focused workflow aligned to: Automates critical appraisal and quality assessment for research papers by analyzing text against established methodological standards (such as risk of bias tools, quality checklists, or reporting guidelines) and synthesizing a structured evaluation report. Use when you need to assess the methodological quality, internal validity, or reporting completeness of any type of study—including RCTs, observational studies, systematic reviews, qualitative research, or diagnostic accuracy studies
The main issue in this stress run was: The script execution path completed successfully for the documented case.
Basic 26/40|Specialized 60/60|Total 86/100
✅A1The quality-assessment output structure matches the documented deliverable
❌A2The script execution path completed successfully for the documented case
✅A3The output stays fully within the documented skill boundary
✅A4The response quality is acceptable for the documented path
Pass rate: 3 / 4
Medical Task Total86.2 / 100
Key Strengths
- Primary routing is Data Analysis with execution mode B
- Static quality score is 88/100 and dynamic average is 72.6/100
- Assertions and command execution outcomes are recorded per input for human review
- Execution verification summary: Script verification 1/1; adjustment=5. extract_pdf.py: OK