What did robert boyle call the groups of atoms that he studied?

He called them molecules.

Explanation:
Boyle is an Irish chemist who is credited with replacing alchemy with modern chemistry. He is credited with defining the inverse relation between the absolute pressure and volume of an ideal gas, known as Boyle's Law.
Boyle called groups of atoms as molecules. It was the work of other researchers that distinguished between atoms and molecules many years later.