We wanted to follow up our plan for affected users and provide a post-mortem that should help the community understand how something like this could happen and what we are doing to make sure that it doesn’t happen again.
Issue Summary
As part of our risk management contributions to Moonwell Apollo, Gauntlet submits updates to parameters in the protocol that allow the community to balance risk and capital efficiency. This is done via governance. The main issue was that an incorrect parameter was submitted in a proposal, and then that proposal was passed by the community without anyone noticing the error. Our post here covers what happened, as well as how it affected users.
Why did this happen
-
We have extensive testing, but it was insufficient
- We have a variety of tests - fork tests, unit testing. We rely on simulation to test the semantic validity of the proposal.
-
The values tested in sim were different than the value submitted in the final proposal
- There is a manual step here to configure sims and proposals that allowed human error to invalidate testing we had done
-
Gauntlet proposals were not tested by other Moonwell contributors.
What we are doing
-
We are adding more testing that will compare parameters to previous values as well as known good ranges to help catch the types of errors in the future
- We are changing our processes to remove manual steps that add risk
-
We will continue to
- Increase code review coverage
- Work closely with the community (including Moonwell contributors) to enable testing and ensure testing is completed
Closing
We strongly believe that increased testing and the right internal processes will drive the chances of incorrect parameters being submitted to zero. However we will work with the rest of the community to ensure that proposals are tested. While we do not intend to rely on this, it should serve as an additional backstop to ensure proposals to governance are correct and clear to voters.