Greg Wilson on Nostr: Matthews & Nagappan 2024: "Design choices made by LLM-based test generators prevent ...
Matthews & Nagappan 2024: "Design choices made by LLM-based test generators prevent them from finding bugs"
https://arxiv.org/abs/2412.14137 Shows that LLM-generated tests can fail to detect bugs and, more alarmingly, how their design can worsen the situation by validating bugs in the generated test suite and rejecting bug-revealing tests. #nwit
Published at
2024-12-19 11:06:36Event JSON
{
"id": "f25dcd3f308581fe0b7b89f87b252014cf52a23a8834479d21e575b349426cf1",
"pubkey": "471ec8585d6c2fd5f911c7a8444add9942c4b8e37365db2a1cb87c5e13c0df8f",
"created_at": 1734606396,
"kind": 1,
"tags": [
[
"t",
"nwit"
],
[
"proxy",
"https://mastodon.social/users/gvwilson/statuses/113679164772305136",
"activitypub"
]
],
"content": "Matthews \u0026 Nagappan 2024: \"Design choices made by LLM-based test generators prevent them from finding bugs\" https://arxiv.org/abs/2412.14137 Shows that LLM-generated tests can fail to detect bugs and, more alarmingly, how their design can worsen the situation by validating bugs in the generated test suite and rejecting bug-revealing tests. #nwit",
"sig": "a0234acdac7265e131635d60fe06a869ec698e311505aedde6da1553b4c9a223751f5cb2a10f0e8e22f62f4f8e6d8b50107fe522dd0ac90bcef9f89ee90d557e"
}
