Clevedon Civic Society has suggested that the old cottage part of the hospital, which dates back to the 16th century, be considered for listing to protect it for future generations.
The move comes as NHS North Somerset has submitted outline plans to demolish the Cottage Hospital and build 14 new homes on the site.
The current hospital was built in 1874 and has 20 beds split over two levels, but the NHS says it no longer meets the needs of the community.
NHS North Somerset has already secured outline planning permission for a new £15 million, 3,500sq metre hospital on the former Kwik Save supermarket site at Mill Cross.
The older part of the hospital, which now houses the accident and emergency department, was originally a barn and formed part of Highdale Farm. It was converted into a hospital by the Elton family and opened in May 1875.
Clevedon Civic Society conservation and planning committee chairman Bryan Osborne said: “We have put forward a suggestion to North Somerset Council conservation officer that this building should be considered for listing.
“It is a very old building although it has been greatly altered over the years.
“We wanted to put forward a suggestion it is grade one listed because of the building’s historical connection with the Elton family (wealthy benefactors involved in building many of the town’s landmarks including the pier) and with Highdale Farm, which is already listed.
“We would like to protect the building against demolition and see it incorporated into the new plans for the site.”
The possibility of whether the building should be listed is now being investigated by North Somerset Council.
Planning consultants working on behalf of the NHS say they believe the old buildings should not be retained as it would directly impact the marketability of the site.
They also said as the barn had been extensively modified over the years it was only of low to medium historic and architectural significance.
The new homes on the hospital site would be two and a half storeys high and designed to complement the existing Victorian street.
They would have two or three bedrooms with additional space in the roof, front and rear garden and there would be 37 parking spaces.
Two separate entrances would be created from Old Street to the development.
Once the outline planning approval is agreed, the site will be sold off to the highest bidder.
The Seasons office block, which sits alongside the hospital, will be sold separately for business use.
The NHS is bound by government legislation to obtain the best price for the site when it become surplus to requirements and plough any funds from the sale back into health services.
The new hospital, expected to open in 2013, will have a 24-bed inpatient unit on the first floor, a counselling suite, administration block and a therapy unit and garden.
It will also have a minor injuries department, which will open 8am to 8pm, seven days a week, and provide accommodation for the out-of-hours GP service.
There will also be an imaging unit, an outpatients’ wing, a pedestrian piazza at the entrance to the building and a therapy garden.
The current hospital employs about 90 staff but the new development will create 35 extra jobs, with 125 people expected to work there.