I have found, with the new tracker:
If there is a pokestop nearby, it will show all the Mon around it. If there are three or less, you get three in a sightings under it. Never had more than three sightings if any showed at a stop. If there is no stop within range it will show up to 9 Pokemon, as before. I have never had 9 show, but had 7 one time so the bottom row is valid.
It does have some bugs, or maybe just the way it is supposed to work. I went Onix hunting with two phones. Many never showed up on the tracking, it is random which 9 appear. Some showed on one phone not the other. Some said “Pokemon ran” or something, indicating it had despawned. The other phone still had it selected and showed it as being there, and it was. It will also say Pokemon Ran if you go out of range of it, as in walking away from it. That is not a bug, just the range limitation.
The selected Pokemon appears as the first selection on the main screen, highlighted. It may or may not appear when you open the Nearby popup. This might mean there is a limit of how many can be stored on the nearby tracker. It is more than 9 because a selected one might not appear, but still be selected. Something like 12 maybe. So that some are tracked, but not showing. But there is a limit because one phone would say it ran away, but the other would still have it selected.