linux multi-port server design in c/c++

Discussion in 'Linux Networking' started by noloader, Nov 26, 2013.

  1. noloader

    noloader Guest

    Hi All,

    Is anyone aware of a discussion or treatment of a multi-port threaded server on modern Linux boxes written in c/c++? Multi-port would mean, for example, ports 80 and 443.

    Contemporary Linux and Unix post-date W. Richard Steven's Advanced Programming in the Unix Environment and Unix Network Programming, so my bibles are not helpful. Googling is returning the Comp Sci 101 examples with a single server/single port.

    Specifically, I'm interested in discussions of:

    (1) multiple processes servicing a single port and thread the accept()
    (2) single process with multiple threads for each port and thread the accept()
    (3) the most efficient way to wait and handle connections
    (4) how to safely shutdown a multi-port, multi-threaded server

    I'm fairly certain fork/exec is no longer recommended, especially since critical libraries like OpenSSL are not fork-safe in a multi-threaded environment.

    I also think item (3) can be handled with libevent, but I'm also interestedin a proper discussion in this setting so I can avoid the dependency. (I try to minimize or eliminate dependencies).

    Finally, my apologies if this is not a programming group. Google's Usenet interface simply sucks. We can't navigate group hierarchies any longer underthe new and improved design. And searching for "comp.os.linux groups" returned one result of comp.os.linux.misc.

    Thanks in advance.
    noloader, Nov 26, 2013
    1. Advertisements

  2. noloader

    David Brown Guest

    This is not really a programming group - but you might get some ideas
    here. I don't know what the ideal group would be - comp.lang.c and
    comp.lang.c++ are more about the language than applications.

    There is no such language as "C/C++". A server like this would be
    structured totally differently in C and in C++. Don't mix these up -
    pick one language.

    Also, is there a particular reason to pick C or C++ ? There are lots of
    other languages suitable for server design, many with frameworks that
    will give you most of what you want here without any effort. Python
    with the "twisted" framework springs to mind (since I've used it myself)
    as being very efficient at handling multiple ports with multiple
    connections, and it takes care of all the details of resources. There
    are many other frameworks for many other languages which could make the
    job a lot easier.
    David Brown, Nov 26, 2013
    1. Advertisements

  3. noloader

    Chris Davies Guest

    Google offers many seemingly-relevant results for a search of
    "multithreaded socket select".

    There may well be discussions on these if you throw the keyword "apache"
    in to the mix.

    Chris Davies, Nov 26, 2013
  4. noloader

    noloader Guest

    Thanks Chris. Try adding "multi-port" (or one of its derivatives).
    Yeah, I'm trying to avoid studying Apache source code.

    noloader, Nov 26, 2013
  5. noloader

    noloader Guest

    Oh, sorry about that.

    I'll try another group or Stack Overflow.
    Yes. We are looking to maximize performance, and I have over 15 years experience with the languages.
    Yeah, I thought about Python. There are a few problems with it. (1) I can read Python, but I don't write it very well (this is a big problem in practice ;). (2) Python is interpreted and not native, so I think C/C++ will havesomewhat better performance. (3) Python makes some really dumb decisions out of the box, and I don't know where all the devils lie (cf, this 2007 bug:

    noloader, Nov 26, 2013
  6. noloader

    David Brown Guest

    No apology needed - posting here is fine too. I'm just warning you that
    you won't get all the answers!

    However, if you are posting to a Usenet group, try to use a proper
    newsreader - Google Groups makes a hideous mess out of posts.
    A lot of people think C and C++ are the answer to "maximum performance".
    Usually they are wrong.

    Once the shock of that wild generalisation has blown over, I'll explain
    my reasoning. C and C++ (along with other compiled languages like Ada,
    Fortran, ocaml, etc.) give you the smallest and fastest code at the
    detail level. But the speed of a system like this will depend on its
    structure, its threading, queuing, synchronisation, its use of blocking
    and non-blocking sockets, the algorithms used to track large data
    blocks, etc. These things are not dependent on the quality of a
    compiler or interpreter - they are dependent on the quality of the
    design. Higher level languages - such as Python - make it easier to
    make good designs than more detail-oriented lower level languages.
    Ergo, such code is often faster when written using a higher level language.

    Secondly, much of the work in a Python program is handled by the
    libraries, rather than being written by the developer. And much of that
    code, especially where there are /real/ performance gains to be made, is
    written in C. And since this code is written by people who have been
    willing to spend a lot of time and effort making efficient and
    well-tested code, such code is normally faster than you would write
    yourself (I don't know how good a programmer you are, but I assume you
    have other priorities such as development time and costs that limit your
    efforts in making the fastest possible code).

    So in theory, C and C++ should give the fastest server code. In
    practice, using the right existing framework greatly overwhelms any
    speed benefits from these languages.

    And assuming you are getting paid for your work here, you are /not/
    looking to maximize performance. You are looking for /good enough/
    performance - anything more is a waste of your employer's or client's
    money. That's worth keeping in mind, especially because employers and
    clients often /say/ they want "maximum performance". You are always
    looking for a balance between higher performance, and the costs of
    getting there.

    Of course, experience with the languages is another matter :)

    Python is a relatively easy language to work with (at least for simpler
    code), and your aim here is to use an existing framework for most of the

    If this is a big enough project, learning Python and writing the system
    in Python may be faster than writing it all in C++.

    (And Python is just an example - there are many others, such as Ruby on

    Python is byte-compiled and runs in a virtual machine. Performance is
    good for some things, poor for others - it is very fast for manipulating
    data, handling strings and text, etc., but slow for basic arithmetic.
    And for performance you can always use PyPy, which has JIT compilation
    to give near-native performance.

    That particular case is a "debatable" decision, rather than a "dumb"
    decision. And there will /always/ be such issues with whatever
    frameworks or libraries you choose to use - even if you write
    /everything/ yourself, you will feel some decisions in the Linux kernel
    are "dumb".

    But it is a good reason for picking well-known and well-tested languages
    and frameworks, rather than picking the latest and greatest trends.
    Python /is/ such a well-known and well-tested language, and Twisted is
    such a framework - they are massively popular.

    Just as a hint, Google is a major backer of Python development, and they
    use it for a lot of their servers and other software.

    I certainly don't mean to say that you should switch to Python for this
    project. But I do think that at this stage, you should keep your
    options open and look at the bigger picture, so that you can make an
    informed and justifiable decision.


    David Brown, Nov 27, 2013
  7. noloader

    Aragorn Guest

    Not to mention that Google Groups is also one of the favorite posting
    media for spammers, which is why most of the regulars filter out
    anything coming in through Google Groups, which reduces a bona fide
    poster's chances of getting a useful reply.
    Aragorn, Nov 27, 2013
  8. noloader

    David Brown Guest

    Most of the spam I see on newsgroups is from direct posters - but I
    don't see much. Maybe my newsserver filters spam better than yours.

    But as long as there are people who filter out Google Groups posts, your
    point is well made.
    David Brown, Nov 27, 2013
  9. noloader

    Jorgen Grahn Guest

    Yes they are.

    Comp.unix.programmer seems like the right place to me, but it has
    rather few posters (often with rather strong and unusual opinions).
    I don't think so: unless you jump on the C++11 train, what determines
    the high-level structure is the Linux APIs you use, not if you use C
    or C++. At least in my experience. Of course C++ will let you do a
    lot of things safer and more elegantly, but not this part.

    Jorgen Grahn, Dec 8, 2013
  10. noloader

    Jorgen Grahn Guest

    Sure about that? There have been updated editions of his books. I
    still use the ones from the 1990s, plus the man pages. epoll(7) is
    the only major newer interface I use ... on the other hand I don't do
    threads and maybe the state of the art has changed more there.
    Yes; the quality of such search results is low. I still see new code
    being written by cargo-culting them; it's more primitive than what
    you'd get from Stevens' 20 years old edition.
    Depending on what you want to do, fork is still the right choice ...

    I'd like to be able to answer your questions (or read the answers) but
    sorry, no. I'm playing with one or two projects of my own, but my
    design is based on a single process, epoll and non-blocking sockets.
    I expect it to be I/O-bound rather than CPU-bound, and I dislike and
    distrust threading ... so I don't think it's useful to you.

    Jorgen Grahn, Dec 8, 2013
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.