Everything described above is useful, but I will add what we use
A Profiler mirror.facebook.net/facebook/xhprof/doc.html
Really very best thing. This extension, which is easily controlled from PHP. Of course the results looks not so nice like xdebug + kcachegrind but there is a very necessary feature for you. The profiling results of the various pages can be grigirovich one average report. It is possible in production to profilerate, say 0.01% of all requests, and then look grigorevny report. It will be clear what f-tsiya brake. With xdebug it is possible not to guess and profilerate not the URLs.
Just answer the questions, Yes I tried to turn it on under load and a strong overhead is not noticed.
For SQL fiercely suggest www.maatkit.org/doc/mk-query-digest.html
Able to use as a source of SHOW PROCESSLIST, and binlog even analyze the network traffic through tcpdump. The most useful — able to unites queries into logical classes with accuracy to parameters.