|Sir Isaac Newton|
Portrait: Godfrey Kneller (1689)
Newton's belief in the particulate nature was supported by his optical experiments. His view was that light is a stream of corpuscles, and in order to explain the fact that some material is transparent he assumed that some of the corpuscles pass through matter without encountering the particles of which it is composed. Matter must therefore consist mainly of empty space, and he made estimates of the size of the particles. His conclusion was that the particles must be extremely small that if all the particles in the solar system came together the total volume would be that of a nut. Later Joseph Priestly (1733-1804), in his "Disquisitions Relating to Matter and Spirit" (1777), first used the expression "matter in a nutshell" to describe Newton's ideas.*Newton's reasoning lacked a fuller understanding of how light interacts with matter. Light will resonate with matter if it encounters an energy match--but otherwise it passes on through or reflects away. Charged "corpuscles" like an alpha particle are another matter. Ernest Rutherford used them when he proposed atoms to be mostly space--cf. his famous "backscattering experiments." But electrons are not empty space--anyone who has seen electron densities as revealed in X-ray crystallography knows that atoms and molecules fill space quite densely.
*Keith J. Laidler, The World Of Physical Chemistry, Oxford University Press: New York, 1993.
There's more on the origin of the expression "in a nutshell" here from Ask Yahoo. Others trace it back to Shakespeare.