Spice does not currently support multiple connections to the same qemu instance. So anybody who will connect to the same host and port can simply take over your session. You can eliminate this problem by using ticketing or SSL.
Ticketing is a simple authentication system which enables you to set simple tickets to a vm. Client has to authentificate before the connection can be established. See the spice option password in the following example.
To set a Spice password for a virtual machine, go to this machine details in virt-manager, and then click on the "Display Spice" item in the left pane, and enter the ticket you want to use in the "Password" field.
All you need to do is to append a passwd attribute to the Spice graphics node for your virtual machine:
<graphics type='spice' passwd='mysecretpassword'/>
Adding a ticket with QEMU involves a slight modification of the -spice parametr used when running QEMU:
-spice port=3001,password=mysecretpassword
When you start the client as usual, if ticketing was enabled on the host, remote-viewer will pop up a window asking for a password before starting the Spice session. It won't be established if an incorrect ticket was passed to the client.
You might have figured out that passing tickets as a commandline option isn't very safe. It's not safe as everybody with access to the host can read it from the output of ps(1). To prevent this, the ticket can be also set by using the qemu console command spice._set_ticket.