Tuesday, July 21, 2015

Problems with Inter-thread communication in Monkey on RTEMS

The problem started with monkey using pipe for communication between threads. The kqueue support in RTEMS libbsd does not go well with the RTEMS pipe implementation, and we get an error of the form :
 [Error] kevent: Invalid argument, errno=22 at 
The errno 22 for kqueue is EINVAL, that says that the specified time limit or filter is invalid.
A bit of tracing revealed that _mk_event_channel_create calls pipe and passes those file descriptor to kevent where it causes the error. Further, it happens after mk_sched_launch_worker_loop (in mk_server/mk_scheduler.c) is called

Notes on Inter-thread communication in Monkey. 

From what I have understood about monkey's architecture, it uses threads ( which in its terminology are called workers) to process connections. There are two modes in which monkey works. One is the loop balancer mode. The loop balancer runs in the main process context and it waits on an event queue for arrival of connections. Once a connection arrives it decides which worker may handle the connection. It then registers the accepted file descriptor on the worker queue. 
mk_server_loop_balancer is detailed in mk_server/mk_server.c : https://github.com/monkey/monkey/blob/master/mk_server/mk_server.c#L252

The mk_server_loop_balancer is called from the mk_server_loop function : https://github.com/monkey/monkey/blob/master/mk_server/mk_server.c#L461

This is the function primarily responsible for waking up the workers. The line:
n = write(sched_list[i].signal_channel_w, &val, sizeof(val));
In the function writes to the channel so that the worker reads from the channel and starts functioning. 
The signal_channel_w is basically one end of the pipe called in _mk_event_channel_create

There is another mode for the functioning of monkey, the REUSEPORT mode, which I am having some trouble understanding.
Whenever I try over-riding the configuration by setting :

allow_shared_sockets == MK_TRUE;
mk_config->scheduler_mode = MK_SCHEDULER_REUSEPORT; 

in mk_bin/monkey.c , the simulation in qemu exits, probably because of  mk_config_listen_check_busy(mk_config) returns MK_TRUE.  This is a recent observation and needs to be investigated further.

 In the present case, most of the work happens in mk_server_loop_balancer. 

Efforts for correcting the behavior

Various methods were tried for correcting the behavior. 
1. A self made file implementation. 
   A pipe implementation was created where it created and then opened a file in write mode, and also in read mode ( thus generating two file descriptors) ,  and passed those two file descriptors as the ends of the file. It gave the same EINVAL error and I later found out that this was very similar to the way pipe was implemented in RTEMS,  so this didn't let us move ahead. 

2. A different pipe using sockets. 
  The code of this attempt goes as :

static int myport=1025;
 int mypipe(int *__fd){
        struct sockaddr_in saddr;
        int rv;
        int lfd;

        lfd = socket(PF_INET, SOCK_STREAM, 0);
        if(lfd < 0) return -1;

        memset(&saddr, 0, sizeof(saddr));
        saddr.sin_family = AF_INET;
        saddr.sin_port = htons(myport++);
        saddr.sin_addr.s_addr = htonl(INADDR_ANY);

        rv = bind(lfd, (const struct sockaddr *) &saddr, sizeof(saddr));
        rv = listen(lfd, 3);

return 0;
(this is also the code present in my current github repo).

What I was trying to do was to create a sort of channel for communication using sockets. Where we could listen to any address on the server, and it could use the socket as a medium for communication between threads. I tried duplicating the sockets and passing them as two ends of the pipe. This saved us from the EINVAL kqueue gave ( and I think, that refining this may as well give a way out ). But the problem was that often it gave an error that socket was not connected. 

3. A SocketPair implementation.
 I was not aware of how socketpair was implemented in FreeBSD. Though, I found a socketpair implementation online, 


This is not my code so I wasn't sure I could use it, so I didn't talk about it. But this gets our project closest to working. This socketpair_tcp implementation works very fine on linux ( just replacing the pipe call to socketpair_tcp works nicely). But on RTEMS, after the file descriptor has been made non-blocking ( line: 74 , socketpair.c ) the connect always sets the errno as EINPROGRESS, and even after waiting for half an hour ( I have tried ) this status doesn't change on the descriptor, which is really confusing.  
Now, interesting thing is, that not making the descriptors non-blocking, or making them non-blocking after connect has been called on them, ( line:77 , socketpair.c ) makes our code works fine. It also outputs 
mk_server_worker_loop() Worker 0 started (SIGNAL_START) 
 like it normally should. But the difference in output for normal execution, and our methods 2 and 3 are different ( discussed below ).

4. Enabling socketpair in rtems-libbsd.
This was the only method I had no idea about but this was infact one of the first methods I tried. There is a certain lack of information on how one would import a system call from freeBSD to rtems.
I tried to study the code and found that the definitions of socketpair were present in rtems-libbsd/freebsd/sys/kern/uipc_syscalls.c . But they were under the preprocessor #ifndef __rtems__ . The most logical thing appeared to me was to change the #ifndef to #ifdef __rtems__ , and then recompile libbsd, resolving each error that occurs ( I am using waf , as Makefile doesn't work ). But for that I had to enable ( #ifndef __rtems__ to #ifdef __rtems__ ) lots of sections, for example the struct thread.  
It didn't appear to me that my way of approach was right so I didn't proceed too far ahead in this direction.

Some issues that may demand attention

Monkey Trace output.

One issue that comes with every implementation that I tried ( whether it was socketpair or it was my own implementations ) was that the trace output generated by monkey on rtems and the one generated by monkey on linux was different. 
After the program starts waiting for events in mk_server_loop_balancer, I telnet to the ip address  ( ,in my case ).  Which it successfully connects.

'Connected to
'Escape character is '^]'.

Then I issue the following commands to get its 'homepage' :
GET / HTTP/1.1

There is activity on the trace in monkey, but no reply gets sent.
When monkey is running normally on linux, I get an HTML dump of the homepage in my telnet window. ( which is the normal behavior).  

This is an issue I discussed with Eduardo few days ago. He suggested me to try
 some things which I did and they didn't work. Later I forgot to reply ( which was very unprofessional on my part).

The output in rtems's case is :

mk_server/mk_scheduler.c:566] mk_sched_event_read() [FD 8] Connection Handler / read
mk_server/mk_http.c:1433] mk_http_sched_read() [FD 128789] Create HTTP session
mk_server/mk_http.c:278] mk_http_handler_read() MAX REQUEST SIZE: 32768
mk_server/mk_http.c:323] mk_http_handler_read() [FD 128789] read 16
mk_server/mk_http.c:353] mk_http_handler_read() [FD 8] Retry total bytes: 16
mk_server/mk_http.c:1470] mk_http_sched_read() [FD 128789] HTTP_PARSER_PENDING
mk_server/mk_stream.c:88 ] mk_channel_write() [CH 8] CHANNEL_EMPTY
mk_server/mk_scheduler.c:566] mk_sched_event_read() [FD 8] Connection Handler / read
mk_server/mk_http.c:278] mk_http_handler_read() MAX REQUEST SIZE: 32768
mk_server/mk_http.c:323] mk_http_handler_read() [FD 128789] read 16
mk_server/mk_http.c:353] mk_http_handler_read() [FD 8] Retry total bytes: 16
mk_server/mk_http.c:1470] mk_http_sched_read() [FD 128789] HTTP_PARSER_PENDING
mk_server/mk_stream.c:88 ] mk_channel_write() [CH 8] CHANNEL_EMPTY
mk_server/mk_scheduler.c:566] mk_sched_event_read() [FD 8] Connection Handler / read
mk_server/mk_http.c:278] mk_http_handler_read() MAX REQUEST SIZE: 32768
mk_server/mk_http.c:323] mk_http_handler_read() [FD 128789] read 2
mk_server/mk_http.c:353] mk_http_handler_read() [FD 8] Retry total bytes: 2
mk_server/mk_http.c:1456] mk_http_sched_read() [FD 128789] HTTP_PARSER_OK
mk_server/mk_http.c:704] mk_http_init() [FD 8] HTTP Protocol Init, session 0x30fe50
mk_server/mk_http.c:842] mk_http_init() [FD 8] STAGE_30 returned -1
mk_server/mk_header.c:393] mk_header_set_http_status() Set HTTP status = 200
mk_server/mk_socket.c:40 ] mk_socket_set_cork_flag() Socket, set Cork Flag FD 8 to ON
mk_server/mk_http.c:223] mk_http_request_prepare() [FD 8] HTTP Init returning 0
mk_server/mk_stream.c:105] mk_channel_write() [CH 8] STREAM_IOV, wrote 171 bytes
I dug into the output further and sorted the issue to one problem that I cannot understand, the mk_channel_write() [CH 8] CHANNEL_EMPTY line which comes in rtems, but doesn't on linux.

Secondly, I also observed that there is a warning Warning] Could not set TCP_FASTOPEN that comes on rtems. This is also a recent observation that needs to be investigated.

 Alternative I am working on at the moment

The method used by monkey for inter-thread communication looks very similar to message queues. So presently I am working to re-write the whole event channel subsystem using message queues. As mentioned in the mail, I have started working on this yesterday afternoon, and hopefully will finish it by today afternoon/today night. 


Saturday, May 23, 2015

Setting up RTEMS development environment

Setting up the development environment involves :

RTEMS Tools From Source

RTEMS Source Builder (RSB) is a tool that is used to create a tool-chain that is used to compile RTEMS applications. Usually ( as is my case ) it will be a cross development environment. I am developing for the i386 processor on an x86_64 Debian Linux system.
RSB creates the tool-chain, debuggers among other things that are used to compile and test RTEMS ( in all called a tool-set ).
the whole development system in my case has been set up in


1. Setup a development work space:
$ cd
$ mkdir -p development/rtems/src
$ cd development/rtems/src
2. Clone the RTEMS Source Builder into the 'src' directory
$ git clone git://git.rtems.org/rtems-source-builder.git 
$ cd rtems-source-builder
3. Checking if the host is set up correctly,using the sb-check tool

$ ./source-builder/sb-check 

if the output is "RTEMS Source Builder environment is ok" then we are good to go. Else we will have to install the required package, specified in the output of sb-check.
( Note: dev package of python2.7 would have to be installed to compile gdb ) 

4. Build Set

Now getting to create the build set of rtems4.11 for i386.  We create a environment variable PREFIX_COMPILER which will be the default location where our compiler and tools will be installed.

$ cd rtems
$ export PREFIX_COMPILER=$HOME/development/rtems/4.11 
$ ../source-builder/sb-set-builder --log=1-pc386.txt --prefix=$PREFIX_COMPILER 4.11/rtems-i386 
It will download and compile the required tools and install them in the directory in $PREFIX_COMPILER. 
After this, the tools have to be put in front of the path variable so that bash searches them first. 


Configure and Build RTEMS

1. Clone rtems from the rtems git repository in the     ~/development/rtems directory . 

$ cd ~/development/rtems
$ git clone git://git.rtems.org/rtems.git
$ cd rtems
$ ./bootstrap

2. RTEMS builds out of the source tree. So we create a directory where it will be built.

$ cd ~/development/rtems
$ mkdir pc386
3. Make a directory where it is to be installed and create an environment variable pointing to it

$ mkdir bsp-install 
$ export PREFIX=$HOME/development/rtems/bsp-install 

4. Configure RTEMS.

$ cd pc386 
$ ../rtems/configure --target=i386-rtems4.11 \
--prefix=$PREFIX --enable-tests=samples \
--enable-networking --enable-rtemsbsp=pc386
$ make 
$ make install

Building Sample Applications 

1. Clone examples-v2 from the repository
$ git clone git://git.rtems.org/examples-v2.git examples-v2
2. Create an RTEMS_MAKEFILE_PATH environment variable 
$ export RTEMS_MAKEFILE_PATH=$PREFIX/i386-rtems4.11/pc386
3. Compile 
$ cd examples-v2
$ make
Note: The environment variables created here could (and should, for convenience) be put into the bashrc file especially the RTEMS_MAKEFILE_PATH, so that it avoids the trouble of creating them again and again.

1. RTEMS Source Builder :
2. RTEMS Git repository : 




    Introductions and the Summer of Code

    Soon it will be a month since my proposal got accepted by RTEMS into Google Summer of Code. Somehow or the other, writing a blog was something I was not able to do. Though, now that I have started, it will remain active and up to date in the time to come. 

    Now with brief introductions. 

    Me : I am Sujay Raj. A student who recently completed the 4th year of a 5 year integrated master's course. My course is Mathematics and Computing, and I consider myself as a part-time mathematician, and a full time programmer. In the time I am not working, I read Manga and novels, watch Anime or think about philosophical aspects of life. I am a GNU/Linux evangelist, and have helped many people switch from other operating systems (usually MS Windows) to a flavor of Linux.

    GSoC : The Google Summer of Code (GSoC) is a program in which Google awards stipends to all students who successfully complete a requested free and open-source software coding project during the summer (which I think is better, compared to Google Code Jam as it will hopefully help people focus on creating something, instead of giving all their time to competitive programming).

    RTEMS : Real-Time Executive for Multiprocessor Systems, is a free and open source real-time operating system designed for embedded systems. It is the parent organization under which I am to work the whole summer. I first stumbled across RTEMS while searching around the web for study materials for my Operating Systems course. Though at that time I had no idea that I would end up working with them. :)

    Monkey : Monkey is a lightweight and powerful web server and development stack for GNU/Linux. It forms the gist of my summer of code project, as well as, the title of the blog. My project is "Porting Monkey HTTP server to RTEMS", which is self-explanatory (though in the later part I will be dealing with some reorganization of RTEMS's network stack too).  My mentors for GSoC are Chris Johns, Joel Sherril ( both from RTEMS ) and Eduardo Silva (from the Monkey Project ). 

     The title of the blog :  Mon-Key , phonetically in Hindi means "Mind's" and Shantih stands for "peace". So, Monkey shantih will phonetically in Hindi will mean Mind's Peace.

    Now being done with introductions, my next post will cover setting up RTEMS development environment and its networking. 

    References : 

    Google Summer of Code : 
    RTEMS : 
    RTMES Wiki: 
    Monkey HTTP server :