Thanks a lot for the answer, Michael (and sorry for the slow response)!
So, if I understand what you're saying correctly, I'm seeing this behavior because wal-e keeps fetching wal files from s3 regardless of this trigger_file, and these fetched wal files are in pg_wal (or pg_xlog), therefore Postgres just tries to restore whatever available in pg_wal before the failover. Or, even if there is no file in pg_wal, it still tries to fetch from the "archive" (s3).
In other words, if I would like to do "immediate failover" (and do not care about WAL files available in archive or in pg_wal), I should be tweaking restore_command so that no further fetching/restoring happens.
Is it... accurate?
Thanks,
Keiko