The main problem I think is the key exchange and user authentication. In your case, I think, to authenticate all authorized clients need either a trusted third party for storing certificates; or the possession of some shared secret information in advance of communication over an insecure channel; or direct access to each other, without the server, to minimize the possibility of compromise of the communication channel.
As I see it, although, of course, I could be wrong.
1. A trusted third party needs to publish certificates or public keys to clients. You can use ssl.
2. General information, this is the symmetric encryption key or an asymmetric private key or any other secret information, proof of the possession of which will enable to authenticate the client.
3. Alternative communication channel which will allow the exchange to authorized users with keys without intermediaries in the form of servers.
And then the user simply encrypts the information and places on the server.