Locals have complained about the site in Banham off New Buckenham Road, arguing the movement of lorries had caused lots of noise and disturbance.
Waveney Construction Landscaping put up several shipping containers, which were used as offices, a workshop and storage.
READ MORE: ‘Act now!’ – Anger in flooded town with more rain on the way
Steve Ballam of Waveney Construction Landscaping argued the containers blended in with the surroundings and didn’t affect the view of the landscape and that the structures were only temporary.
He also said their landscaping work and planting of wildflowers has improved the amount of wildlife in the area, creating six and a half acres of wildlife reserve.
There was strong opposition from locals, who worried allowing it would set a precedent for businesses to build on similar sites which could impact the countryside in the future.
At a planning committee meeting, Breckland councillors voted unanimously to refuse the retrospective application – throwing the future of the business in doubt.
The business may be forced to tear down the buildings on the site following the decision.