The goal of optimization is to make your program run faster. While we'd like the compilation to be fast too, that's not something you control with optimization switches.
It may be that your application is not CPU or memory-bound, which limits the effect of optimization. You can use code profiling tools such as prof, or if you have a Windows PC available, you can use Intel's VTune performance analyzer with the data collector run on Linux.
If your application is waiting for disk I/O to complete, or is being limited by memory throughput, there may not be much a compiler can do to help you. A good analyzer will tell you where your program is spending its time.
Steve