Work has begun on replacing the section of wall at St Joseph's church facing Hill Street.

Father Greg Bellamy, the parish priest for the Parish of St Mary and St Joseph, said the work was being done to preserve the heritage.

"We want to protect this part of the heritage not only for the Catholic community but for the whole community of Orange," he said.

"This is the first step toward caring for St Joseph's into the future."

Other works planned include repairing the building's stained-glass windows, fixing the belltower, repairing the roof, improving heating and rewiring the church's electrical system.

Parish operations manager Michael O'Mara said the wall had been leaning out for the past 14 years and was progressively getting worse.

"There was a risk the wall would fall over. It had to be fixed," he said.

However Mr O'Mara said they were surprised to find the structure of the wall was weaker than they thought, once work had started.

He said they had thought it was supported by rock but instead there was only layers of dirt and clay behind it.

Mr O'Mara said when workman started it took little effort for a machine to bring the whole wall down.

He said many of the bricks were damaged.

"We were hoping with the bricks they could be cleaned and re-used but they are cracked, two out of three," he said.

However, he said they had found out the same bricks were still made today and considered using replacement bricks.

Instead, following heritage requests, a mix of new and old bricks will be used in the new wall.

Mr O'Mara said work was two weeks into an expected 6-8 week project.

"We would like to have this fixed while the kids are on school holidays," he said.

Mr O'Mara said they were hoping to have restoration work completed for their 150th anniversary in 2021.

He said a key part was restoring the stained-glass windows.

"We've removed nine and we have an application before council to take out a further 10," he said.

"We've got a few windows restored but they are still held in security.

"We can't put them back in until we fix the bell tower and the roof."