Protocol Design

esm

87100Total Score
Core Capability
84 / 100
Functional Suitability
11 / 12
Reliability
9 / 12
Performance & Context
7 / 8
Agent Usability
14 / 16
Human Usability
8 / 8
Security
10 / 12
Maintainability
9 / 12
Agent-Specific
16 / 20
Medical Task
20 / 20 Passed
93Designing novel proteins with desired properties by generating sequences (optionally conditioned on structure/function) using ESM3
4/4
89Completing or editing sequences (e.g., filling masked residues, generating variants) for protein engineering workflows
4/4
87ESM3 multimodal generation across *sequence*, *structure*, and *function* tracks
4/4
87Local inference (e.g., esm3-sm-open-v1) and cloud inference via Forge (e.g., esm3-medium-2024-08, esm3-large-2024-03)
4/4
87End-to-end case for ESM3 multimodal generation across *sequence*, *structure*, and *function* tracks
4/4

Veto GatesRequired pass for any deployment consideration

Skill Veto✓ All 4 gates passed
✓
Operational Stability
System remains stable across varied inputs and edge cases
PASS
✓
Structural Consistency
Output structure conforms to expected skill contract format
PASS
✓
Result Determinism
Equivalent inputs produce semantically equivalent outputs
PASS
✓
System Security
No prompt injection, data leakage, or unsafe tool use detected
PASS
Research Veto✅ PASS — Applicable
DimensionResultDetail
Scientific IntegrityPASSThe archived evaluation treated outputs as protocol guidance to be tested later, not as validated experimental findings.
Practice BoundariesPASSThe package remained on the planning side of the boundary and did not cross into clinical or diagnostic advice.
Methodological GroundPASSThe legacy review kept the package aligned with its named analysis library, data structure, or processing workflow.
Code UsabilityN/AThe package is evaluated primarily as a structured deliverable rather than an executable scientific code workflow.

Core Capability84 / 100 — 8 Categories

Functional Suitability
The archived deduction in functional suitability traces back to: Improve stress-case output rigor. Stress and boundary scenarios show weaker consistency
11 / 12
92%
Reliability
Related legacy finding for esm: Improve stress-case output rigor. Stress and boundary scenarios show weaker consistency
9 / 12
75%
Performance & Context
Performance-context scoring suggests the package could compress some of its longer planning paths more cleanly.
7 / 8
88%
Agent Usability
Agent usability was strong, though the package could make its decision points even easier to follow at first read.
14 / 16
88%
Human Usability
No point loss was recorded for human usability in the legacy audit.
8 / 8
100%
Security
Security scored well, though the archived review still left some room to make boundary language even more explicit.
10 / 12
83%
Maintainability
Maintainability held up, but a little more consolidation or clearer packaging would likely close the remaining gap.
9 / 12
75%
Agent-Specific
Agent specific was softened by the legacy issue 'Improve stress-case output rigor'. Stress and boundary scenarios show weaker consistency
16 / 20
80%
Core Capability Total84 / 100

Medical TaskExecution Average: 88.6 / 100 — Assertions: 20/20 Passed

93
Canonical
Designing novel proteins with desired properties by generating sequences (optionally conditioned on structure/function) using ESM3
4/4 ✓
89
Variant A
Completing or editing sequences (e.g., filling masked residues, generating variants) for protein engineering workflows
4/4 ✓
87
Edge
ESM3 multimodal generation across *sequence*, *structure*, and *function* tracks
4/4 ✓
87
Variant B
Local inference (e.g., esm3-sm-open-v1) and cloud inference via Forge (e.g., esm3-medium-2024-08, esm3-large-2024-03)
4/4 ✓
87
Stress
End-to-end case for ESM3 multimodal generation across *sequence*, *structure*, and *function* tracks
4/4 ✓
93
Canonical✅ Pass
Designing novel proteins with desired properties by generating sequences (optionally conditioned on structure/function) using ESM3

Designing novel proteins with desired properties by generating... stayed in planning mode and returned a bounded design deliverable without relying on a runnable script.

Basic 36/40|Specialized 57/60|Total 93/100
✅A1The esm output structure matches the documented deliverable
✅A2The instruction path remains actionable for the documented case
✅A3The output stays fully within the documented skill boundary
✅A4The response quality is acceptable for the documented path
Pass rate: 4 / 4
89
Variant A✅ Pass
Completing or editing sequences (e.g., filling masked residues, generating variants) for protein engineering workflows

This variant a case remained a study-design support path, not a code-driven execution run.

Basic 34/40|Specialized 55/60|Total 89/100
✅A1The esm output structure matches the documented deliverable
✅A2The instruction path remains actionable for the documented case
✅A3The output stays fully within the documented skill boundary
✅A4The response quality is acceptable for the documented path
Pass rate: 4 / 4
87
Edge✅ Pass
ESM3 multimodal generation across *sequence*, *structure*, and *function* tracks

The archived run treated ESM3 multimodal generation across *sequence*, *structure*, and... as a protocol-design path rather than an executable workflow.

Basic 33/40|Specialized 54/60|Total 87/100
✅A1The esm output structure matches the documented deliverable
✅A2The instruction path remains actionable for the documented case
✅A3The output stays fully within the documented skill boundary
✅A4The response quality is acceptable for the documented path
Pass rate: 4 / 4
87
Variant B✅ Pass
Local inference (e.g., esm3-sm-open-v1) and cloud inference via Forge (e.g., esm3-medium-2024-08, esm3-large-2024-03)

This variant b case remained a study-design support path, not a code-driven execution run.

Basic 32/40|Specialized 55/60|Total 87/100
✅A1The esm output structure matches the documented deliverable
✅A2The instruction path remains actionable for the documented case
✅A3The output stays fully within the documented skill boundary
✅A4The response quality is acceptable for the documented path
Pass rate: 4 / 4
87
Stress✅ Pass
End-to-end case for ESM3 multimodal generation across *sequence*, *structure*, and *function* tracks

The archived run treated End-to-end case for ESM3 multimodal generation across *sequence*,... as a protocol-design path rather than an executable workflow.

Basic 29/40|Specialized 58/60|Total 87/100
✅A1The esm output structure matches the documented deliverable
✅A2The instruction path remains actionable for the documented case
✅A3The output stays fully within the documented skill boundary
✅A4The response quality is acceptable for the documented path
Pass rate: 4 / 4
Medical Task Total88.6 / 100

Key Strengths

  • Primary routing is Protocol Design with execution mode A
  • Static quality score is 84/100 and dynamic average is 80.6/100
  • Assertions and command execution outcomes are recorded per input for human review
  • Execution verification summary: No script verification was applicable