Not: był?
I would guess the subject of the sentence is “to”
In this structure, the verb matches the person (unlike in English, where you say it wasn’t us etc.). So you will say “to nie byłem ja”, “to nie byliśmy my”, “to był on”, “to byli oni”, and so on.
I’m not sure if there’s a logical way to explain this. My gut feeling is that “to” in this particular context is somehow grammatically different from “to” in sentences like “To są moi rodzice” (“These are my parents.”). But maybe it’s just an exception or a linguistic “accident”.
I can’t think of any similar structures that would behave this way, apart from equivalent phrases in the present or future tense (e.g. “To nie są oni” — “It isn’t them”.)