Replanting camellia

I have a camillia in a pot for about 4 years and nothing much has happened. The odd flower here and there. I am thinking of replanting out of the pot and into the garden. Good idea or not?