The James being sick could be explained 1 of two ways. 1, he did in fact catch some bug, but more likely, he was dead when they took him in 27 years ago, and since its the barn keeping him alive, the longer out the weaker he gets. That's why Howard let him out, he knows he has to come back.

And there's no guarantee that time passes inside the Barn at the same rate as outside. Or maybe he did know Audrey had left for the first time 3 years ago and was hanging out with Howard.

That said, it did seem like they painted themselves into a corner a bit with the Barn plot.