I believe I was told that the smoke from dryer lint can be somewhat toxic (if you have a lot of cotton clothes) by Andy Wayne... possibly.
I think it was on this forum.
I just use cotton balls now.

especially if i would be in a situation where I may only have a spark that I need to nurture into a flame, as I always end up breathing in some smoke and getting in my face.

I'm sure it's not that strong, but still somewhat toxic IIRC.