Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B (phind.com)

Article summary

A fine-tuned CodeLlama-34B model has achieved better results than GPT-4 on the HumanEval benchmark. The model used sampling with temperature=0.1, and reproduction details are available on the Huggingface model card. The achievement is seen as a testament to the power of open source models. However, some commenters note that the HumanEval benchmark has limitations, such as a small dataset and lack of real-world software engineering problems.

Main themes

• AI model benchmarking
• open source models
• programming language support
• fine-tuning and training
• benchmark limitations
• practical applications
• AI acceleration and concerns

What commenters say

• The use of different generation methodologies, such as sampling versus greedy decoding, can affect the comparability of results between models.
• Fine-tuning models for specific programming languages, such as Pascal, could be beneficial but may require significant resources and data curation.
• The cost of fine-tuning models can be relatively low, with estimates ranging from $30 to $3000.
• Some commenters argue that the HumanEval benchmark is not a comprehensive measure of a model's coding abilities, while others see it as a useful starting point.
• There is a need for more diverse and representative benchmarks to evaluate the performance of coding models.
• The potential for open source models to accelerate progress in AI research is significant, but also raises concerns about test set contamination and overfitting.
• The practical applications of coding models may not require out-of-distribution generalization, but rather a good enough representation of existing code.
• The rapid advancement of AI technology is seen as both exciting and alarming by different commenters.