This house occupies a site associated within the medieval Manor of Riffhams. The latter was a satellite of nearby Graces Manor and may have been named after Richer and John de Refham who lived in the village in the early 14th century.
The current house is based on a 16th century timber house that was enlarged and refaced in brick in the 18th century. In the early 19th century, Riffhams was owned by J.R.Spencer Phillips who built a new house, now called New Riffhams, and let the old house to tenants. By the middle of the-century, Old Riffhams was the home of three labourers and their families. Later, it was occupied by the estate gamekeeper, Charles Smoothy, a keen ornithologist and taxidermist, who donated his collections of birds to Chelmsford Museum. In 1912, it was the home of Reverend George Woolley and his family, including his son Sir Leonard, who would direct archaeological excavations at Ur in the 1920s and 30s. From 1919, the house was occupied by Mr Herbert Paterson, who lived there for over thirty years and left many black and white sketches of the Danbury and the Little Baddow area.