Continuity mistake: When Betty walks down the stairs to calm the Hulk down the camera angle from the front shows her hair blowing in the wind, but the camera angle from the back doesn't.
Continuity mistake: When Bruce Banner is submerged in the holding tank at the desert lab, he has hair on his chest. But when he changed back from the Hulk at Betty's cabin, his chest was smooth.
Continuity mistake: When Betty takes Bruce back to his childhood home, she sits on a swing prior to entering the house. The swing is moving as the go into the building but is stationery a few moments later (it wouldn't have had time to come to a complete standstill).
Revealing mistake: When Hulk rips the turret off the tank and swings it around in the desert has a solid base. This is correct for a model but not for a real tank which has a hollow turret for the tank captain to stand up from the bottom of the tank through the turret and look out the top.
Answer: There are varying interpretations of Bruce's memory in the comics. The most common one is that Bruce has vague memories of what happens while he is the Hulk (i.e. fear, hunger, images), but because the Hulk's brain is less "evolved" he can't remember specific information, like where he went or what he did. The converse is also true: The Hulk has vague memories of Bruce's life as well...he recognizes people that Bruce knows but doesn't know anything about them.