You could just leave them wired up and twisted with the white signal wire, with no connection on the end near the FC. That should provide some noise reduction. If you have any issues with noise, you could solder the 4 unconnected grounds together to one short wire going to the battery ground pad.