I would say you won't notice any difference in CPU performance but without a doubt Adreno 306 is quite a bit less powerful than Adreno 320.
Also make sure to get the 2gb ram model because from my personal experience with Moto E, 1gb ram is just not enough when opening up lots of apps simultaneously.