The simple answer is, baby squirrels don't leave the nest until they are fully furred and can survive on their own so, without seeing the mother right next to the babies, they all look about the same size.
Most babies leave the nest in April or May. At this point the babies are fluffy and fat but the parents have exhausted their winter fat and are beginning to shed their winter fur, so look relatively small. A second litter of babies may leave the nest around September. At this point the parents have begun putting on fat and winter fur, so the apparent size difference can be greater. Because of this, it may be easier to identify babies born late in the year.
Although baby squirrels have been recorded in almost every month of the year, these two litters, early spring and late summer, are the norm. Typically the early spring babies have the highest survival rate, especially in areas where it snows, since a small squirrel has to expend more energy than a large one to stay warm and find food.
I managed to take a few pictures of a baby and a young adult male that were foraging near each other in the same park. Unfortunately, tree squirrels aren’t very social so I couldn’t get any useful pictures of them near each other, but they found my pen interesting so there is some scale. The pen is about 16cm long.
A baby grey squirrel.
An adult grey squirrel from the same population as the baby in the previous photograph.
Another view of the adult. Note the more “mature” features.
Even with the specimens in-hand, assessing age can be somewhat qualitative but when the babies are very young, they are simply more cute than the rest of the population.
If you're a squirrel watcher, like me, I hope you take the time to record your observations at projectsquirrel.org . Your data, combined with that of others around the country, helps us understand more about squirrels and about the nature in your neighborhood!