Nuneaton is currently experiencing a battle – a polite one, but a battle nonetheless – about what counts as heritage. The outbuildings of the Georgian farmhouse where the novelist George Eliot spent her first 21 years are about to be pulled down. Whitbread, which owns them – it runs the main house as a cheerful family pub – says they don’t count as part of the literary fabric of Britain. The George Eliot Fellowship, of which I am a vice-president, say that they do. The planning meeting to decide the fate of these listed buildings is in early July.

At first glance, these huts don’t have swagger value. They are vernacular Warwickshire from the early 19th century, red-bricked and roofed with tiles made from one of the hyper-local quarries that date back to medieval times. More importantly from a literary-historical point of view, these unassuming buildings were crucial in forming the imagination of the girl who became one of Britain’s greatest novelists. As the daughter of a tenant farmer, Eliot spent her youth making butter and feeding the poultry in these sheds, known formally as the farm’s “offices”. And it is these hard-working spaces, just as much as the flagstoned and mullioned main house, that went straight into her best-loved novels. If you’ve ever wondered what the dairy looked like where Hetty Sorrel is seduced in Adam Bede – one of the most famous scenes in English literature – then go and stare at these modest structures before they disappear.

The question of which buildings or parts of buildings matter and which do not is one that has changed in the past 20 years. Visit a grand National Trust property these days, and you’ll find that it is in the kitchens and the stables that people like to linger. Upstairs, in the grand hall, there is a limit to how many details you can take in about the Louis Quinze daybeds and curlicued what-nots. And as for all those portraits hanging over the staircase, they soon start to merge. But cooking, eating and getting around – these are activities that hold their charge across the centuries.

The recognition that it is the demotic and the diurnal that matter to us when thinking about the past is fairly recent. Until the 1970s we were expected to visit historic houses as a way of absorbing public, national narratives. What the Duke of Devonshire did at Chatsworth or Winston Churchill did at Chartwell mattered because it told a certain kind of story about Britain and Britishness. History was made in the study, the library and the muniments room and not in the attic bedrooms or the garden sheds. But the rise of social history as an academic discipline, and its trickle down to the school curriculum, means that it is perfectly reasonable to want to know what it was like to milk a cow by hand in the frosty dawn or pick your way across a farm courtyard in the squelching mud.

That’s why this summer families will be flocking to places such as the Ironbridge Gorge museums in Ironbridge, Shropshire, where you can poke around small, unlovely cottage workshops and try your hand at weaving, tile decorating, metalwork. It’s why, for many an enthusiastic home cook, the idea of a perfect weekend break might be signing up to learn how to make cheese the artisanal way in the old stable block on the Welbeck Estate in the middle of Sherwood Forest.

For the lazier among us, there’s the altogether more passive pleasure of nipping into the old dairy on a day out at a National Trust property and gorging on a cream tea or sauntering out to the kitchen garden to marvel at the runner beans. Braggart buildings are all very well but, ultimately, it is the untidy corners, the garden sheds and unpretending workshops that draw us imaginatively into the past and hold us there.

• Follow Comment is Free on Twitter: @commentisfree