Can't search invisible (un-filtered) contacts

Hi there, with my prevous phone (Xperia x10 mini pro with Android 2.3), in Contacts, I had selected that only "My Contacts" was shown but I could do a search and find contacts in my other (invisible/un-filtered) groups. With my Z1 Compact (Android 5.1) it only searches those contacts that are shown. I don't want to make all my contacts visible since this makes my phonebook cumbersome, but I'd like to be able to search them all. Can anyone help with this?