Is it possible not to specify a password for PHPMailer?

No, you cannot.

And to send emails to the specified email, you must specify the username and password of the account where you want to receive letters.

And you enter the access account from which to send the letter. And not Vice versa.

Possible, but not for all mail providers.
For example, in Офис365 used identification by IP: add the IP of the server in the list right boys and can send on behalf of any organization's account without a password.
Such a feature might have in the business package of Google (I haven't used it ever, so not going to say anything).

For free postal no password definitely not. But you don't need your account password (if you configured two — factor authentication- if not impossible). Use the app password.

Added

And to send emails to the specified email, you must specify the username and password of the account where you want to receive letters

It makes more sense to use AddReplyTo.