Hello :)
The people who were there with Sir Henry Cooper and taking care of him seemed concerned about him as it was extremely cold for him. For people wandering round it may not have felt so cold, but sitting there still and at 75 years old I'm sure he felt it much more than most others.
He was gone for around an hour and fifteen minutes for lunch and to warm up and in all honesty, he needed that time to warm up, I myself was even concerned about him.
I think it's just unfortunate timing that you were ready to leave when Sir Henry had a much needed break.
:)