Seaford Head Community College (SHCC) has delivered courses to hundreds of adults in the local community under a franchise agreement with Sussex Downs.
But earlier this year, employees were informed that the adult education centre could close.
Since then users and residents have called for the facility to remain at the site in Arundel Road.
SeeAbility, a charity based in Seaford which caters for people with visual impairment and additional disabilities, has residents that attend the courses at SHCC.
Speaking of the potential closure of the centre Fiona Grafton, registered manager of Applewood, said, "This would be totally devastating for people living at Applewood and Barclay House, SeeAbility's other residential home in Seaford, where five people with multiple disabilities attend the college.
"It would be very difficult to arrange transport to other colleges in Lewes or Eastbourne. We are hoping that it will remain open."
The adult education centre offers four distinct strands of learning including mainstream, NVQs in the workplace, Skills for Life and courses for adults with learning disabilities.
Following a review by East Sussex County Council of the demand for adult education and costs of the service, it has been decided the current model cannot continue and it has been proposed that the centre will close.
The letter sent out to employees read, "Discussions are underway to see if another provider can, working in partnership with the Community College, ensure that provision is still available to local residents."
One resident who met with Mr Baker said if the facility went it would leave 'a big void' for the
town's people and that the classes served both retired people as well as those with disabilities.
They added, "It is a known fact that as one gets 'younger' the brain needs to be kept stimulated and the many courses at Seaford Head helps to keep us mentally fit and hopefully useful citizens in the many volunteers jobs we all do."
Seaford MP Norman Baker added, "I don't understand it with a town of 25,000 why this facility cannot be sustained. It's very regrettable."
A spokesperson for East Sussex County Council and the Community College said, "No decision has been taken to close the adult learning function at Seaford Head Community College. The current service is not financially viable so there is a recommendation that the community college stops offering adult education itself by the end of March next year.
"This issue is currently the subject of consultation with staff and unions which will end on December 7. The community college's governors will then consider the outcome of the consultation.
"An essential part of the consultation process is allowing time for staff to come forward with viable options for provision. The county council and community college cannot consult other stakeholders at this stage as the current consultation is a statutory process for staff only. "However, further communication and discussions with a wider group of people will be considered once this statutory consultation is complete. We will continue to do all we can to minimize the impact on individual students and staff."