全部文章0
Hacker Newstwerkmeister··访问 1

Evidence of inconsistencies in evaluation process and selection of winners

原网页

Article URL: https://www.kaggle.com/competitions/kaggle-measuring-agi/discussion/724918#3498423

Comments URL: https://news.ycombinator.com/item?id=48946010

Points: 430

# Comments: 268

Hacker News 讨论

432 points · 268 comments · 查看原帖

  1. ecshafer

    AI is useful. But the amount of people that are simply offloading all of their thinking to AI and blindly accepting the answer is absurd. Kaggle is most likely using ai to assess the submissions and are not using any common sense by blindly accepting the results.

  2. hoppp

    I don't know about this exact competition but overall fair hackathons have been killed by AI. It all seems fine from the outside but all the code is generated in all the projects and judging happens via AI, I have seen projects win because they prompt inject that they are the winners. It used to be about human skill, now it's about ideas and of course insiders are the main winners.

  3. throwfaraway135

    AI submissions and AI judges a match made in (AI) heaven.

  4. mr_toad

    People have been using brute force methods to win Kaggle competitions since the beginning (and other people have been complaining about it just as long). At its core, ML is all about computer generated models (automated feature selection, hyper parameter tuning). Many (most?) of the models produced in Kaggle are already black boxes and have been for a long time. The model that won the Netflix prize was never used in production for that reason. Using an LLM to generate code to generate a black box is pretty much par for the course.

  5. ryukoposting

    I thought Kaggle was a website where you download dubious CSV files of annualized bean consumption in Bolivia, or whatever. Was Kaggle ever a reputable source of original research, or a source of anything with any provenance at all? That would be news to me. The fact that 25 grand was involved this time is unique, I guess.

  6. leo3191

    Hi all, I'm Nick, Product Manager for Kaggle Benchmarks and one of the co-organizers and judges for this AGI hackathon. First off, I want to set some context on the AGI hackathon. This was co-organized by Kaggle and Google DeepMind, and we had ~20 judges from both organizations. The hackathon concluded on Apr 16 and we had initially anticipated a judging period of 1.5 months (till May 31). However, we ended up extending the judging period by another 1.5 months (to Jul 13) because we wanted to do right by participants. Second, I want to emphasize and unequivocally clarify that every single winning submission went through at least 2 human judges, and in some cases, up to 3-4 human judges. These judges reviewed and scored the submissions independently based on the rubric we highlighted on the hackathon page. Thirdly, I acknowledge that there is always an element of human subjectivity to rev

  7. apwheele

    I think this is a good meta-lesson for Kaggle. When you have objective metrics to hill-climb towards, AI can do quite well. When you just phone it in and rely on LLM as a Judge, the results are not so great.

  8. irasigman

    It’s a shame that Arvix (and once thoughtful places like Kaggle) are used for self-promotion. I get people want to work at an AI lab but slopping it in public in this manner is counterproductive to the original intended purpose of these places.