What the canonical tag does
When the same content is reachable at several URLs (with and without www, with tracking parameters, with a trailing slash), the canonical tag tells search engines which one to index and credit. It is a strong hint, not a command.
Most pages should have a self-referencing canonical: the tag names the page's own clean URL. Problems start when it points at a redirect, a 404, another host, or the home page from every page.
Questions
Should every page have a canonical tag?
Yes, a self-referencing one is the safe default.
Can a canonical point to another domain?
Yes, for syndicated content. Pointing your own pages elsewhere by mistake removes them from search.
What if there are two canonical tags?
Google may ignore both. Keep exactly one.
Does it check the HTTP header too?
Yes, a Link header with rel=canonical is read as well.