One problem I run into sometimes is that multiple things on a server are trying to access the same disk. This is pretty unavoidable, and often you want the “default,” which is for them to share the disk. If I have two webserver threads, they’re equal.
But sometimes this isn’t the case. A while back I had to make a backup of the local 600GB partition that one of our NFS servers was exporting, and the rsync was really killing NFS performance. I was able to “nice it down” a bit and make it work.
dm-ionice is another interesting option that I just came across, though. It does “bandwidth quotas” for disks. I can see a lot of places where this might be handy. I’m curious about exactly how it works and the full implications, but it’s a feature that I think is needed in many places but not normally available.