They are rarely held up as examples of refined, gentle behaviour.
Now scientists have found evidence that cavemen really were the violent and competitive knuckleheads depicted in movies and cartoons.
A study of fossilised remains suggests that our ancient ancestors had far higher levels of the male s*x hormone testosterone than people living today.
If the findings are confirmed, it means they were more aggressive and promiscuous than modern men - and that tens of thousands of years of evolution have had a civilising influence on the human race.
The study was carried out by British and Canadian scientists who worked out the testosterone levels of extinct apes and ancestors by looking at the length of fossilised finger bones.
Past studies have shown that exposure to testosterone in the womb can make humans and apes more aggressive and more promiscuous.
The same hormones also alter the way babies develop physically in the womb.
Boys exposed to high levels tend to have a longer ring finger relative to their index finger, while in males exposed to lower levels, the two fingers tend to be a similar length.
The scientists worked out the ratio of the two finger lengths for a range of ancient hominins - or ancient members of the family tree - including four Neanderthals and an early modern human from 70,000 years ago.
They also studied the finger lengths of an ape-like ancestor called Australopithecus afarensis which lived three million years ago and an even older species called Ardipithecus ramidus from 4.4 million years ago.
Although the researchers only looked at a handful of specimens, the findings suggest that Neanderthals and early humans were more testosterone fuelled than modern men.
The results, published in the journal Proceedings of the Royal Society B, also found that Australopithecus - a species that walked on two legs and whose best known fossil is the female nicknamed Lucy - may have been monogamous.
Ardipithecus, however, was more promiscuous - and may have behaved like modern great apes.
Dr Emma Nelson, of Liverpool University, said: 'It is believed that prenatal androgens (male s*x hormones) affect the genes responsible for the development of the fingers, toes and the reproductive system.
'We have recently shown that promiscuous primate species have low index to ring finger ratios, while monogamous species have high ratios.
'We used this information to estimate the social behaviour of extinct apes and hominins. Although the fossil record is limited for this period, and more fossils are needed to confirm our findings, this method could prove to be an exciting new way of understanding how our social behaviour has evolved.'
Dr Susanne Shultz, from the Institute of Cognitive and Evolutionary Anthropology at the University of Oxford added: 'Social behaviours are notoriously difficult to identify in the fossil record.
'Developing novel approaches, such as finger ratios, can help inform the current debate surrounding the social systems of the earliest human ancestors.'