Solution
Just go to the app info of the app (Apps>Google Drive) then delete all the used storage on the device (Click on Storage>Clear Storage). Since your files haven't been uploaded they are stil on your device and they are located in the files of Drive. Clearing cache won't solve this *Clear...