Drive through the affluent Kent countryside at this time of year and the whole area seems to be turning red. The listed tile-hung cottages are mellow, virginia creeper is turning scarlet against white weatherboard farmhouse walls, the apples are ripening in the orchards. And this year the rubicund faces of those in the know about planning matters are growing just a little bit more choleric.

The one thing guaranteed to rouse the semi-mythical Disgusteds of Tunbridge Wells to fury are planning decisions. It is an irony therefore that the MP for this town in the bluest part of Britain is Greg Clark, the minister charged with selling the government’s planning shake up, which shifts the presumption in favour of development even as the coalition insists that the green belt will be protected and localism in decision-making will rule. Clark is understandably struggling to square that circle.

Housing statistics show that planning permission is not the main obstacle to house building – capital funding for builders and mortgages for buyers are: 80% of residential building applications were granted last year, more on appeal.

The suspicion is that this is a political fight. Chancellor George Osborne wants to show the government is kick-starting recovery and communities secretary Eric Pickles is happy to take on local authorities and their officialdoms. That puts them in direct conflict with Tories in Clark’s constituency.

“Greg’s heart’s in the right place, but I believe the government has lost its way on this. As a political problem it has the potential to blow up in their faces,” says Sean Holden, a Tory councillor in the small nearby town of Cranbrook, who is a former national party spokesman. “In the heartlands, I don’t believe the majority of Conservative voters want housing estates going up.

“It stands directly in the path of the government’s localism bill about decisions being made at a local level. People are not comfortable with the presumption in favour of development. I would like an entitlement for people to say they do not want their community developed in this way. That should be a reason for refusal: that we don’t want it.”

It is a sentiment shared by many of his colleagues on the overwhelmingly Tory Tunbridge Wells borough council, though officially they are fully behind the government’s plans.

Linda Hall, a local councillor who was quoted in last weekend’s Sunday Times saying supporters were appalled, found herself berated by the MP himself at a drinks party later that day.

“He was very upset with me. Cameron had rung him up apparently and given him a hard time. He said if journalists contact me, I should let him know. But I think it is my job to speak my mind. That’s why I was elected. I was only passing on what people are saying.

“Greg attended a meeting of local party members last week and although most were respectful he did get an earful with one or two saying there would not be a housing problem if we didn’t have any immigrants.”

With only the slightest hint of glee, Alan Bullion, a leading local Lib Dem, says: “I think Greg has shot himself in the foot here. There are arguments, but you do not want to trample all over the planning laws at the expense of conservation – that is doing your best to piss off your own constituents.”

Clark is not one of the government’s Bullingdon Boys. The son of working class parents, he was educated at a Middlesbrough comprehensive, before going on to Cambridge, a doctorate at the LSE, then starting politically in the SDP.

But he only has to take a short stroll from his townhouse by Tunbridge Wells common – no danger of developers spoiling the view there as it’s protected by an act of parliament – past Thackeray’s old home and the Conservative Club to the old Kent and Sussex hospital, to experience the ire of local residents.

It is closing next week – replaced by a £230m hospital, initiated by the last government, on the edge of town – and the plan is to build 380 homes on the old site. The scheme is already provoking protests and complaints from locals who say, rightly, the town’s infrastructure won’t stand it: the roads are already clogged and there aren’t enough school places.

The council is also struggling with the fallout of a controversial regeneration scheme it entered into with the developers John Laing, that roused widespread local anger last year when it emerged that the Tory leadership wanted to sell off the grand art-deco 1930s civic centre – incorporating council offices, theatre, police station and library – to build a shopping centre instead. The row cost the last council leader Roy Bullock, who doubled as Clark’s election agent, his position. That is precisely the sort of presumption in favour of development that locals fear.

Interviewed by Channel 4 News last weekend on the common, with a backdrop of some of the town’s more agreeable Georgian and Victorian villas – properties which sell for upwards of £1m – Clark said: “You can see behind me buildings which are attractive to residents and attract visitors. That is my ambition, to get better standards of building that are sustainable, with the consent of local communities, rather than having development forced on them. We want places that people would love to live in, rather than being mean and dispiriting in design.” There is no doubt more affordable housing is needed: expecting young couples to find 25% deposits when even local flats cost £100,000 and terraces £200,000 is just impossible.

“What does affordable mean?” says Holden. “We’ve got 286 new houses planned in Cranbrook, which means 600 more people at least in a town of 6,000, with no change in the infrastructure and you can bet residents’ children won’t be able to afford them. Affordable is a politician’s buzz word and it irks me.”

Had Clark walked five minutes in the opposite direction from his home to the Victorian Italianate villas of Hungershall Park (“remarkably unaffected by later intrusions, villas of … no particular merit. The delightful thing is that [they] face each other across a gently-sloping pasture field,” says the Pevsner guide) he could have consulted Mike Sander, a local resident and former chief executive of nearby Crawley council.

“The government’s plans are totally unacceptable, they are unnecessary and they won’t work – and that’s based on 40 years’ experience,” Sander says. “Nimbyism is embedded and tinkering with planning policy is not going to make any difference. Most of them don’t have any experience of local government and they spend their time rubbishing it, but there was nothing wrong with planning properly.”

Hall adds: “Greg is such a plausible, lovely person but I think the party could be committing political suicide.

“If you take on the four million members of the National Trust, 90% of whom probably vote Conservative, you are asking for trouble, aren’t you?”