Protocol Design

esm

Toolkit for protein language models (ESM3 for multimodal generative protein design; ESM C for efficient embeddings). Use when you need sequence/structure/function generation or prediction, inverse folding, protein embeddings, or scalable inference via local weights or the Forge API.

87100Total Score

Core Capability

84 / 100

Functional Suitability

11 / 12

Reliability

9 / 12

Performance & Context

7 / 8

Agent Usability

14 / 16

Human Usability

8 / 8

Security

10 / 12

Maintainability

9 / 12

Agent-Specific

16 / 20

Medical Task

20 / 20 Passed

93Designing novel proteins with desired properties by generating sequences (optionally conditioned on structure/function) using ESM3

4/4

89Completing or editing sequences (e.g., filling masked residues, generating variants) for protein engineering workflows

4/4

87ESM3 multimodal generation across *sequence*, *structure*, and *function* tracks

4/4

87Local inference (e.g., esm3-sm-open-v1) and cloud inference via Forge (e.g., esm3-medium-2024-08, esm3-large-2024-03)

4/4

87End-to-end case for ESM3 multimodal generation across *sequence*, *structure*, and *function* tracks

4/4

Veto GatesRequired pass for any deployment consideration

Skill Veto✓ All 4 gates passed

✓

Operational Stability

System remains stable across varied inputs and edge cases

PASS

✓

Structural Consistency

Output structure conforms to expected skill contract format

PASS

✓

Result Determinism

Equivalent inputs produce semantically equivalent outputs

PASS

✓

System Security

No prompt injection, data leakage, or unsafe tool use detected

PASS

Research Veto✅ PASS — Applicable

Dimension	Result	Detail
Scientific Integrity	PASS	The archived evaluation treated outputs as protocol guidance to be tested later, not as validated experimental findings.
Practice Boundaries	PASS	The package remained on the planning side of the boundary and did not cross into clinical or diagnostic advice.
Methodological Ground	PASS	The legacy review kept the package aligned with its named analysis library, data structure, or processing workflow.
Code Usability	N/A	The package is evaluated primarily as a structured deliverable rather than an executable scientific code workflow.

Core Capability84 / 100 — 8 Categories

Functional Suitability

The archived deduction in functional suitability traces back to: Improve stress-case output rigor. Stress and boundary scenarios show weaker consistency

11 / 12

92%

Reliability

Related legacy finding for esm: Improve stress-case output rigor. Stress and boundary scenarios show weaker consistency

9 / 12

75%

Performance & Context

Performance-context scoring suggests the package could compress some of its longer planning paths more cleanly.

7 / 8

88%

Agent Usability

Agent usability was strong, though the package could make its decision points even easier to follow at first read.

14 / 16

88%

Human Usability

No point loss was recorded for human usability in the legacy audit.

8 / 8

100%

Security

Security scored well, though the archived review still left some room to make boundary language even more explicit.

10 / 12

83%

Maintainability

Maintainability held up, but a little more consolidation or clearer packaging would likely close the remaining gap.

9 / 12

75%

Agent-Specific

Agent specific was softened by the legacy issue 'Improve stress-case output rigor'. Stress and boundary scenarios show weaker consistency

16 / 20

80%

Core Capability Total84 / 100

Medical TaskExecution Average: 88.6 / 100 — Assertions: 20/20 Passed

Canonical

Designing novel proteins with desired properties by generating sequences (optionally conditioned on structure/function) using ESM3

4/4 ✓

Variant A

Completing or editing sequences (e.g., filling masked residues, generating variants) for protein engineering workflows

4/4 ✓

Edge

ESM3 multimodal generation across *sequence*, *structure*, and *function* tracks

4/4 ✓

Variant B

Local inference (e.g., esm3-sm-open-v1) and cloud inference via Forge (e.g., esm3-medium-2024-08, esm3-large-2024-03)

4/4 ✓

Stress

End-to-end case for ESM3 multimodal generation across *sequence*, *structure*, and *function* tracks

4/4 ✓

Canonical✅ Pass

Designing novel proteins with desired properties by generating sequences (optionally conditioned on structure/function) using ESM3

Designing novel proteins with desired properties by generating... stayed in planning mode and returned a bounded design deliverable without relying on a runnable script.

Basic 36/40|Specialized 57/60|Total 93/100

✅A1The esm output structure matches the documented deliverable

✅A2The instruction path remains actionable for the documented case

✅A3The output stays fully within the documented skill boundary

✅A4The response quality is acceptable for the documented path

Pass rate: 4 / 4

Variant A✅ Pass

Completing or editing sequences (e.g., filling masked residues, generating variants) for protein engineering workflows

This variant a case remained a study-design support path, not a code-driven execution run.

Basic 34/40|Specialized 55/60|Total 89/100

✅A1The esm output structure matches the documented deliverable

✅A2The instruction path remains actionable for the documented case

✅A3The output stays fully within the documented skill boundary

✅A4The response quality is acceptable for the documented path

Pass rate: 4 / 4

Edge✅ Pass

ESM3 multimodal generation across *sequence*, *structure*, and *function* tracks

The archived run treated ESM3 multimodal generation across *sequence*, *structure*, and... as a protocol-design path rather than an executable workflow.

Basic 33/40|Specialized 54/60|Total 87/100

✅A1The esm output structure matches the documented deliverable

✅A2The instruction path remains actionable for the documented case

✅A3The output stays fully within the documented skill boundary

✅A4The response quality is acceptable for the documented path

Pass rate: 4 / 4

Variant B✅ Pass

Local inference (e.g., esm3-sm-open-v1) and cloud inference via Forge (e.g., esm3-medium-2024-08, esm3-large-2024-03)

This variant b case remained a study-design support path, not a code-driven execution run.

Basic 32/40|Specialized 55/60|Total 87/100

✅A1The esm output structure matches the documented deliverable

✅A2The instruction path remains actionable for the documented case

✅A3The output stays fully within the documented skill boundary

✅A4The response quality is acceptable for the documented path

Pass rate: 4 / 4

Stress✅ Pass

End-to-end case for ESM3 multimodal generation across *sequence*, *structure*, and *function* tracks

The archived run treated End-to-end case for ESM3 multimodal generation across *sequence*,... as a protocol-design path rather than an executable workflow.

Basic 29/40|Specialized 58/60|Total 87/100

✅A1The esm output structure matches the documented deliverable

✅A2The instruction path remains actionable for the documented case

✅A3The output stays fully within the documented skill boundary

✅A4The response quality is acceptable for the documented path

Pass rate: 4 / 4

Medical Task Total88.6 / 100

Key Strengths

Primary routing is Protocol Design with execution mode A
Static quality score is 84/100 and dynamic average is 80.6/100
Assertions and command execution outcomes are recorded per input for human review
Execution verification summary: No script verification was applicable