Sounds like you’re exploring the Lich's Tomb! Once you reach the second floor, all 2's rolled are discarded. The Lich's Tomb banner bounces when this happens to indicate it's the cause for the disappearing dice. You can press on the banner to read the dungeon card and see the effects. The Phoenix's Den also has a similar effect, discarding 3's on the 3rd floor.

If you’ve encountered the Phantom in your dungeon delving adventures, you’ll notice you might have some dice suddenly poof away. During this encounter, any 1’s and 3’s rolled will be discarded, so keep that in mind while planning your strategy!