This is a complex topic, I'll try to explain it as simply as possible.
There is the so called "Saturation Parameter (K)". K is the desired number of active stake pools. From K the maximum amount of ADA delegated to one single pool is calculated. Once a pool exceeds the maximum amount of ADA, it will offer diminishing rewards. (Source https://cardano.org/stake-pool-operation/)
If there would be exactly K stake pools and all ADA would be delegated. All pools would have the maximum amount of ADA delegated (not too much). Therefore that would be the ideal scenario.
In practice, there are more than K stake pools. So the stake pool operators need to try to convince people to delegate to their pool. And you're right, many small pools struggle to get new delegators because people naturally tend to choose a pool that already has more stake delegated.
Recently the minimum fixed fee of stake pools has been decreases from 340 to 170 ADA to make the penalty of staking with a smaller pool less drastic for the delegators. However, this means even less rewards for the pool operators which might not be sustainable for them.
Please also have a look at this related question for more information.