More players = more data being uploaded. And in a peer-to-peer environment, it's up to the clients to do all the uploading.
Home internet connections have pretty terrible upload rates; 1 mb/sec or less in most cases. Reducing player counts reduces the amount of data being transferred in turn reducing the chance for lag.
In my own experience, the only time I'm seeing lag in Forza 4 is during full races with a 16 person lobby. Qualifying is general around 12 players and I've yet to see any crazy lag in qualifying. Also, during the full races, the lag seems to subside after 3 or 4 people quit out and you're left with 12 or so. Like I said, that's just my experience.