So I grew my gingerbread house with castle magic to its biggest size because I wanted to see all the details up close. It's pretty darn cute. But...when I tried to interact with it to see my newly enlarged gobbler munch his house, I couldn't get the 'hit x to interact' message to display no matter how close I got.
So I have done some further testing, and it seems to be only a problem with the gingerbread house. I tried it with some bobbleheads and they worked just fine when grown.