OpenAI GPT 4o ranked as simplest AI model for writing Solidity magnificent contract code by IQ Liam ‘Akiba’ Wright · 2 months in the past · 2 min be taught
BrainDAO launches solidity code generation benchmark tests with SolidityBench.
2 min be taught
Up to this level: Oct. 21, 2024 at 11:25 am UTC
Duvet art/illustration through CryptoSlate. Image entails blended pronounce material which would possibly per chance well comprise AI-generated pronounce material.
SolidityBench by IQ has launched because the predominant leaderboard to effect in thoughts LLMs in Solidity code generation. Accessible on Hugging Faceit introduces two modern benchmarks, NaïveJudge and HumanEval for Solidity, designed to assess and infamous the abilities of AI fashions in generating magnificent contract code.
Developed by IQ’s BrainDAO as half of its coming near near IQ Code suite, SolidityBench serves to refine their very accept as true with EVMind LLMs and study them in opposition to generalist and neighborhood-created fashions. IQ Code objectives to offer AI fashions tailored for generating and auditing magnificent contract code, addressing the increasing need for real and efficient blockchain purposes.
As IQ told CryptoSlateNaïveJudge gives a novel means by tasking LLMs with implementing magnificent contracts in line with detailed specifications derived from audited OpenZeppelin contracts. These contracts provide a gold frequent for correctness and efficiency. The generated code is evaluated in opposition to a reference implementation the use of criteria corresponding to sensible completeness, adherence to Solidity simplest practices and security standards, and optimization efficiency.
The review project leverages advanced LLMsincluding deal of versions of OpenAI’s GPT-4 and Claude 3.5 Sonnet as just code reviewers. They assess the code in line with rigorous criteria, including implementing all key functionalities, going through edge conditions, error administration, supreme syntax utilization, and total code structure and maintainability.
Optimization concerns corresponding to gas efficiency and storage administration are moreover evaluated. Scores vary from 0 to 100, providing a comprehensive analysis across functionality, security, and efficiency, mirroring the complexities of expert magnificent contract style.
Which AI fashions are simplest for solidity magnificent contract style?
Benchmarking results confirmed that OpenAI’s GPT-4o model executed the supreme total internet of 80.05, with a NaïveJudge internet of 72.18 and HumanEval for Solidity toddle rates of 80% at toddle@1 and 92% at toddle@3.
Curiously, more moderen reasoning fashions like OpenAI’s o1-preview and o1-mini had been crushed to the stop position, scoring 77.61 and 75.08, respectively. Units from Anthropic and XAI, including Claude 3.5 Sonnet and grok-2, demonstrated aggressive efficiency with total rankings hovering round 74. Nvidia’s Llama-3.1-Nemotron-70B scored lowest in the stop 10 at 52.54.
