Question about phone memory

[Wakefulpanda]

I notice that after using my phone during the day that gradually the available memory drops from 180mb available (with all tasks killed) to around 100mb available (with all tasks killed) and it starts to run a touch on the sluggish side.

I understand the basics of why this happens but is this supposed to have an effect on the performance of the phone?

I find myself restarting my phone once a day and its becoming a habit for me is it really worth it to restart it?

Thanks