In this study, we systematically investigate the interactions between mobile ions generated from added salts and immobile charges within a sulfobetaine-based polyzwitterionic film in the presence of five salts (KCl, KBr, KSCN, LiCl, and CsCl). The sulfobetaine groups contain quaternary alkylammonium and sulfonate groups, giving the positive and negative charges. The swelling of the zwitterionic film in the presence of different salts is compared with the swelling behavior of a polycationic or polyanionic film containing the same charged groups. For such a comparative study, we design cross-linked terpolymer films with similar thicknesses, cross-link densities, and charge fractions, but with varying charged moieties. While the addition of salt in general leads to a collapse of both cationic and anionic films, the presence of specific types of mobile anions (Cl-, Br-, and SCN-) considerably influences the swelling behavior of polycationic films. We attribute this observation to a different degree of ion-pair formations between the different types of anionic counterions and the immobile cationic quaternary alkylammonium groups in the films where highly polarizable counterions such as SCN- lead to a high degree of ion pairing and less polarizable counterions, such as Cl-, cause a low degree of ion pairing. Conversely, we do not observe any substantial effect of varying the type of cationic counterions (K+, Li+, and Cs+), which we assign to the lack of ion pairing between the weakly polarizable cations and the immobile anionic sulfonate groups in the films. In addition, we observe that the zwitterionic films swell with increasing ionic strength and the degree of swelling is anion dependent, which is in agreement with previous reports on the “antipolyelectrolyte effect”. Herein, we explain this ion-specific swelling behavior with the different cation and anion abilities to form ion pairs with quaternary alkylammonium and sulfonate in the sulfobetaine groups.