In order to address that, I updated my circuit a bit. After doing some searching, it was clear that the standby mode of the Arduino processor isn't good enough; it still draws power because of the voltage regulator. In order to address that, I came up with the following solution:
The 9V battery powers the voltage regulator in one of two ways. When the momentary push button is pushed, the circuit is closed and power is applied. That power will then initialize the processor, which in turn will set a pin to 5V. Setting the pin to 5V will engage the relay, which at that point closes the circuit for the 9V <-> 5V battery also. The momentary button can be released, since the relay is still active.
After some period of no activity, the processor can simply drop the pin to which the relay is connected to low, the relay disengages, and the circuit is broken, which cuts power to the voltage regulator, and as a result to the microprocessor.
As far as I can see now, the only drawback of this design is that the relay also draws a current. The model that I chose clocks in at around 24mA, which compares it to an LED. I could have countered that with a relay that latches, but those were too pricey for me.
All in all-- a solution that appears to work. And, with a timeout set to 2.5 minutes, hopefully less frequent battery replacements.