Now that Internet access is so common, you can do things more spontaneously, but it is still very important to prepare appropriately for your library visits. This means checking that the library in question has the material that you want to use, that the library is indeed open (they close regularly for inventory and cleaning) and that you are allowed to use it. You may need a letter of introduction from your professor or librarian, for example. Be sure to check the library's home page for more information.
The National Diet Library is open to anyone over the age of 20, but since the materials are almost all in closed stacks, you need to be prepared to make requests, wait for service, wait for photocopying, and there are limits on how many materials you can request at one time. There are things that only the NDL has though, and service is much much better than it used to be. It is now possible to put items on hold before you go to help speed up the process.
Prefectural libraries or large municipal libraries often have wonderful collections, particularly if you are working on modern literature or Japanese history, and much of their collection is in open stacks so you can browse and read to your heart's content.They often provide access to databases - such as the full-text newspaper databases - and the reference librarians are quite happy to help out foreigners using their libraries.
Private university libraries - like Waseda - require letters of introduction. Waseda, in particular, has one of the best Japanese literature collections, and is the favorite of foreign scholars. So they are strict about following protocol. Please be sure to follow the rules and prepare accordingly.
National university libraries used to require letters of introduction but do not any longer. You will need to register as a guest to be admitted into the library. Branch libraries - at the University of Tokyo for example, may have different rules. Again, you should read their web pages very carefully.
Please note that you may not be able to access subscription databases if you are visiting a university library.