Because they slow you down. If you have 5 zombies coming toward you and no other survivors are close by, you might kill a couple of them but they will end up overwhelming you by sheer force of numbers.
To clarify: the panic button will drop everything except your crowbar and the weapon you are currently holding
. So switch to your shotgun/colt/other weapon of choice or necessity, drop everything else and run like hell. If you're good, you should be able to stay alive long enough to work your way back to your dropped weapons & ammo by the time you run out of ammo on the weapon you kept