Creating A Simple Python Socket Server
Implement Simple Networking Protocols In Python
You want to have your application send information to a server, and back. You also want to keep your code clean. These requirements sound simple, because in essence they are. However without knowledge of certain strategies, many programmers will end up stuck and confused, on seemingly simple problems.
This tutorial will break down my strategy, which I believe to work very well. If you’ve done this sort of programming before, you’ve already developed your own style. If so, you’re welcome to read and comment on what I teach, however the article itself might not bring anything new for you. With that, let’s get started.
We’re going to be using gevent to make our connections asynchronous, as we don’t want to halt all communications when we get a new client. Gevent comes with some built-in servers, which will come in handy. These servers have things called “handlers”. Handlers are callable objects that will receive connections, and as the name states, handle them. Let’s start creating our own handler class.
Let’s break down the code. First, we create an init and set the conn and addr properties to the respective arguments. Then we call the start method, which is just shorthand for calling main and finish. The reason we use try and finally is because we want to make sure we clean up our mess, even if an error is raised in main. By using finally instead of except, it’ll make sure that after we clean up, the error will be raised so we can see what went wrong. In this example, we’re creating a server that does math, which will makeup most of main.
Here comes he most complex part, receiving information. We’re going to use 2 packages to streamline this. The first is zlib, which comes in the standard library. It’s main purpose is to compress and decompress information, which we utilize to make our payloads as small as possible. The second reason we use it is for the ZLIB_SUFFIX. Using the suffix, we know when the message has ended. The second package we use is msgpack. Msgpack is really useful, because it allows us to turn most python datatypes right into bytes, and using less bytes than json normally would. Going back to the code, we can see we added the new get_msg method. It will continue to add bytes to a buffer until the buffer ends with the ZLIB_PREFIX. If it does end with the suffix, we decompress it, and then use msgpack to load it into a dict. (Msgpack can load more than just dicts, however in this case that’s what we’ll be sending / receiving). In the main method, we use the new method to get a dict which will have an equation for the server to do.
Now we implement the send method to send information back to the client. Inside send is the opposite of get_msg. First we put our answer in a dict, and then use msgpack to turn that into bytes. After that we compress the information using zlib, and send it. We finish the main method, by sending the answer of the math equations to the client.
Now, we finish the finish method by closing the socket. Then we use our handler class to create a gevent server, which as stated before will automagiclly make this asynchronous, and then run that server forever.
That’s it! You can easily build upon this program to implement whatever server you want. Right now, while our program can’t be exploited by the client very easily, we don’t verify the data the client sends us. If you wanted too, you could use something like Cerberus to confirm the data is what we expected, and possibly catch decompression and msgpack loading errors in case the data was create improperly.
I hope you found this tutorial enlightening, be sure to comment any questions you might have about this design, and I’ll do my best to answer them.
this is way over complicated, Third line of the Zen of Python "simple is better than complicated".
@21natzil Instead of importing zlib and msgpack you could have created a simple socket with the socket module. The socket modules has everything you need in it to preform all of the actions you created. For messages, you don't need to compress and decompress you can simply type socket.send('Message')
to recieve a message you connect to the client and use socket.recvfrom((HOST, PORT)) to recv the message. I am sorry if I don't fully understand the features of the code I skimmed over it at 5am.
@UnLuxeSec One problem with that is that the receiving end doesn't know when the message is complete. By using zlib, you not only get lossless compression but also an indicator that lets each side know when the message is complete. And using msgpack is a super fast and efficient way to serialize python objects into bytes. These features remove the need for other code that you would have to add anyways. Hence being more pythonic.
When I run my socketserver (with http.server b/c I'm lazy), I can't access the port on the host it's running at. Any suggestions or tips on dealing with this security feature (presumably)? Any help would be greatly appreciated.
@21natzil Currently, the python server repl is at https://<programname>.<username>.repl.run. But all that provides one with is the log for the program (presumably sandboxed and restricted from opening TCP ports).
How can I change that to https://<programname>.<username>.repl.co, where I can actually send GET requests to it?
Thank you for any tips!
@21natzil Sorry, I think I should've elaborated more on the error part.
to get the host IP address which I tried accessing from.
In this case, the result was
172.18.0.10. I then added
:8080, the port that my program should've opened and sent a GET request.
That screenshot is the error I got.
Thank you for your time!
@21natzil I think this is what I'm doing wrong:
However, I have tried many different hosts and none of them seemed to have worked. For example:
localhost. Maybe I should try using the dynamic public IP of the server, as returned by
Or maybe the command is returning a private IP address?
Or maybe the server is running on a virtual machine that requires special instructions to open a port to public IP addresses? Maybe
sudo permissions are needed?
Or maybe I just shouldn't rely on web services to host for me and I should run the code with my own computer :).
Maybe what I'm doing wrong is that I'm using the wrong repl programming language/type. What should I be using? I'm using Python3, as usual, but should it be something special like Python3 - WebServer? I couldn't find anything as such.
Thank you for any advice!