I found out after doing some searches it's because when you swipe away light flow from recent apps, it gets turned off. You have to reopen light flow and then it will work again. I hope that a fix comes soon for this as it's kinda annoying.
Or you can get even better IEM's cheaper... Future Sonics Atrio M5 with the new MG7 driver. If you use the code CostProg50% at checkout at Atrio? Professional Earphones it takes 50% off the order.