Does anyone struggle with 1996 Retro Nuptse fit/sizing?

I have the Large but it looks very ugly and baggy. The length is perfectly fine but the chest width is crazy big. I like the baginess and boxiness ofc but it’s just too much. Is there a huge difference between the Medium and Large? Thanks.