The former England defender is baffled by the lingering hatred