And Xander's relationship with Buffy also doesn't seem very self-inserty. Part of his defining character in the early seasons is that he has the hots for her, but does he get her? Nope. She's not interested and that's that. It doesn't matter how he feels.

I dunno about that - self-insert doesn't necessarily imply giving that character what they want. I suspect Whedon maybe identified with Xander partly as 'the one with girl friends but not girlfriends'.

Plus ofc letting them get together wouldn't have worked at all with the story they were telling, which speaks to your point about Xander being a layered character. The storytelling & character development is the priority.