We recently encountered a very strange issue where rebooting one of our ESXi 5.0 hosts would cause network disruption (NTI Enviromux devices would drop off the net, older switches would experience packet loss and vLAN reconfig flapping).
Eventually we determined with a packet capture the host was flooding the subnet with ARP requests related to one of its NFS datastores. When we removed this datastore the host could reboot without causing the network disruption. But we needed this datastore to satisfy HA requirement for 2 heartbeat datastores.
The datastores are all provisioned from Netapp with the exports file listing each host's IP address for RW and ROOT access. Upon closer inspection, the problem host's IP was listed only in the RW section of the Netapp exports file. So, while the datastore would show up mounted, ESXi was not given full access it needed for HA heartbeat functionality, and the result was this flood of ARP.
Once the exports file was updated with IP in the ROOT section, the datastore remounted with full permissions and the host stopped exhibiting the problematic ARP, network disruption behavior.
We still have a case open with VMware to determine why ESXi 5.0 Update 1 behaves this way (allowing an NFS mount without the access required for full functionality, plus unleashing an ARP flood)