Diseases from ingesting Cat litter?

I'm sure you all know about the bit of cat litter that gets tracked out of the litter box onto the carpet right in front of the box. Well I was cleaning this up and inadvertently swallowed some of it. I've been incredibly concerned/freaking out about things like Toxoplasmosis and other parasites. Should I be this concerned? Thanks!