What happens when my database is restarted or becomes unreachable? Does my connection pool stick around?

Yes. The pool is independent of its ability to communicate with to the DBMS. All connections in the connection pool may become defunct, but the connection pool still exists.