Problems with Inter-thread communication in Monkey on RTEMS
The problem started with monkey using pipe for communication between threads. The kqueue support in RTEMS
libbsd does not go well with the RTEMS pipe implementation, and we get an error of the form :
[Error] kevent: Invalid argument, errno=22 at
/home/raaj/development/rtems/monkey/mk_core/mk_event_kqueue.c:208
The errno 22 for kqueue is EINVAL, that says that the specified time limit or filter is invalid.
A bit of tracing revealed that _mk_event_channel_create calls pipe and passes those file descriptor to kevent where it causes the error. Further, it happens after mk_sched_launch_worker_loop (in mk_server/mk_scheduler.c) is called.
Notes on Inter-thread communication in Monkey.
From what I have understood about monkey's architecture, it uses threads ( which in its terminology are called workers) to process connections. There are two modes in which monkey works. One is the loop balancer mode. The loop balancer runs in the main process context and it waits on an event queue for arrival of connections. Once a connection arrives it decides which worker may handle the connection. It then registers the accepted file descriptor on the worker queue.
mk_server_loop_balancer is detailed in mk_server/mk_server.c : https://github.com/monkey/monkey/blob/master/mk_server/mk_server.c#L252
The mk_server_loop_balancer is called from the mk_server_loop function : https://github.com/monkey/monkey/blob/master/mk_server/mk_server.c#L461
This is the function primarily responsible for waking up the workers. The line:
n = write(sched_list[i].signal_channel_w, &val, sizeof(val));
In the function writes to the channel so that the worker reads from the channel and starts functioning.
The signal_channel_w is basically one end of the pipe called in _mk_event_channel_create.
There is another mode for the functioning of monkey, the REUSEPORT mode, which I am having some trouble understanding.
Whenever I try over-riding the configuration by setting :
balancing_mode=MK_FALSE;
allow_shared_sockets == MK_TRUE;
mk_config->scheduler_mode = MK_SCHEDULER_REUSEPORT;
in mk_bin/monkey.c , the simulation in qemu exits, probably because of mk_config_listen_check_busy(mk_config) returns MK_TRUE. This is a recent observation and needs to be investigated further.
In the present case, most of the work happens in mk_server_loop_balancer.
Efforts for correcting the behavior
Various methods were tried for correcting the behavior.
1. A self made file implementation.
A pipe implementation was created where it created and then opened a file in write mode, and also in read mode ( thus generating two file descriptors) , and passed those two file descriptors as the ends of the file. It gave the same EINVAL error and I later found out that this was very similar to the way pipe was implemented in RTEMS, so this didn't let us move ahead.
2. A different pipe using sockets.
The code of this attempt goes as :
static int myport=1025;
int mypipe(int *__fd){
struct sockaddr_in saddr;
int rv;
int lfd;
lfd = socket(PF_INET, SOCK_STREAM, 0);
if(lfd < 0) return -1;
memset(&saddr, 0, sizeof(saddr));
saddr.sin_family = AF_INET;
saddr.sin_port = htons(myport++);
saddr.sin_addr.s_addr = htonl(INADDR_ANY);
rv = bind(lfd, (const struct sockaddr *) &saddr, sizeof(saddr));
rv = listen(lfd, 3);
__fd[0]=lfd;
__fd[1]=dup(lfd);
return 0;
}
struct sockaddr_in saddr;
int rv;
int lfd;
lfd = socket(PF_INET, SOCK_STREAM, 0);
if(lfd < 0) return -1;
memset(&saddr, 0, sizeof(saddr));
saddr.sin_family = AF_INET;
saddr.sin_port = htons(myport++);
saddr.sin_addr.s_addr = htonl(INADDR_ANY);
rv = bind(lfd, (const struct sockaddr *) &saddr, sizeof(saddr));
rv = listen(lfd, 3);
__fd[0]=lfd;
__fd[1]=dup(lfd);
return 0;
}
(this is also the code present in my current github repo).
What I was trying to do was to create a sort of channel for communication using sockets. Where we could listen to any address on the server, and it could use the socket as a medium for communication between threads. I tried duplicating the sockets and passing them as two ends of the pipe. This saved us from the EINVAL kqueue gave ( and I think, that refining this may as well give a way out ). But the problem was that often it gave an error that socket was not connected.
3. A SocketPair implementation.
I was not aware of how socketpair was implemented in FreeBSD. Though, I found a socketpair implementation online,
https://www.samba.org/~tridge/junkcode/socketpair.c
This is not my code so I wasn't sure I could use it, so I didn't talk about it. But this gets our project closest to working. This socketpair_tcp implementation works very fine on linux ( just replacing the pipe call to socketpair_tcp works nicely). But on RTEMS, after the file descriptor has been made non-blocking ( line: 74 , socketpair.c ) the connect always sets the errno as EINPROGRESS, and even after waiting for half an hour ( I have tried ) this status doesn't change on the descriptor, which is really confusing.
Now, interesting thing is, that not making the descriptors non-blocking, or making them non-blocking after connect has been called on them, ( line:77 , socketpair.c ) makes our code works fine. It also outputs
mk_server_worker_loop() Worker 0 started (SIGNAL_START)
like it normally should. But the difference in output for normal execution, and our methods 2 and 3 are different ( discussed below ).
4. Enabling socketpair in rtems-libbsd.
This was the only method I had no idea about but this was infact one of the first methods I tried. There is a certain lack of information on how one would import a system call from freeBSD to rtems.
I tried to study the code and found that the definitions of socketpair were present in rtems-libbsd/freebsd/sys/kern/uipc_syscalls.c . But they were under the preprocessor #ifndef __rtems__ . The most logical thing appeared to me was to change the #ifndef to #ifdef __rtems__ , and then recompile libbsd, resolving each error that occurs ( I am using waf , as Makefile doesn't work ). But for that I had to enable ( #ifndef __rtems__ to #ifdef __rtems__ ) lots of sections, for example the struct thread.
It didn't appear to me that my way of approach was right so I didn't proceed too far ahead in this direction.
Some issues that may demand attention
Monkey Trace output.
One issue that comes with every implementation that I tried ( whether it was socketpair or it was my own implementations ) was that the trace output generated by monkey on rtems and the one generated by monkey on linux was different.
After the program starts waiting for events in mk_server_loop_balancer, I telnet to the ip address ( 10.1.2.5:2001 ,in my case ). Which it successfully connects.
'Trying 10.1.2.5...
'Connected to 10.1.2.5.
'Escape character is '^]'.
Then I issue the following commands to get its 'homepage' :
GET / HTTP/1.1
host: 10.1.2.5
<line-feed>
'Trying 10.1.2.5...
'Connected to 10.1.2.5.
'Escape character is '^]'.
Then I issue the following commands to get its 'homepage' :
GET / HTTP/1.1
host: 10.1.2.5
<line-feed>
There is activity on the trace in monkey, but no reply gets sent.
When monkey is running normally on linux, I get an HTML dump of the homepage in my telnet window. ( which is the normal behavior).
This is an issue I discussed with Eduardo few days ago. He suggested me to try
some things which I did and they didn't work. Later I forgot to reply ( which was very unprofessional on my part).
The output in rtems's case is :
mk_server/mk_scheduler.c:566] mk_sched_event_read() [FD 8] Connection Handler / read
mk_server/mk_http.c:1433] mk_http_sched_read() [FD 128789] Create HTTP session
mk_server/mk_http.c:278] mk_http_handler_read() MAX REQUEST SIZE: 32768
mk_server/mk_http.c:323] mk_http_handler_read() [FD 128789] read 16
mk_server/mk_http.c:353] mk_http_handler_read() [FD 8] Retry total bytes: 16
mk_server/mk_http.c:1470] mk_http_sched_read() [FD 128789] HTTP_PARSER_PENDING
mk_server/mk_stream.c:88 ] mk_channel_write() [CH 8] CHANNEL_EMPTY
mk_server/mk_scheduler.c:566] mk_sched_event_read() [FD 8] Connection Handler / read
mk_server/mk_http.c:278] mk_http_handler_read() MAX REQUEST SIZE: 32768
mk_server/mk_http.c:323] mk_http_handler_read() [FD 128789] read 16
mk_server/mk_http.c:353] mk_http_handler_read() [FD 8] Retry total bytes: 16
mk_server/mk_http.c:1470] mk_http_sched_read() [FD 128789] HTTP_PARSER_PENDING
mk_server/mk_stream.c:88 ] mk_channel_write() [CH 8] CHANNEL_EMPTY
mk_server/mk_scheduler.c:566] mk_sched_event_read() [FD 8] Connection Handler / read
mk_server/mk_http.c:278] mk_http_handler_read() MAX REQUEST SIZE: 32768
mk_server/mk_http.c:323] mk_http_handler_read() [FD 128789] read 2
mk_server/mk_http.c:353] mk_http_handler_read() [FD 8] Retry total bytes: 2
mk_server/mk_http.c:1456] mk_http_sched_read() [FD 128789] HTTP_PARSER_OK
mk_server/mk_http.c:704] mk_http_init() [FD 8] HTTP Protocol Init, session 0x30fe50
mk_server/mk_http.c:842] mk_http_init() [FD 8] STAGE_30 returned -1
mk_server/mk_header.c:393] mk_header_set_http_status() Set HTTP status = 200
mk_server/mk_socket.c:40 ] mk_socket_set_cork_flag() Socket, set Cork Flag FD 8 to ON
mk_server/mk_http.c:223] mk_http_request_prepare() [FD 8] HTTP Init returning 0
mk_server/mk_stream.c:105] mk_channel_write() [CH 8] STREAM_IOV, wrote 171 bytes
mk_server/mk_http.c:1433] mk_http_sched_read() [FD 128789] Create HTTP session
mk_server/mk_http.c:278] mk_http_handler_read() MAX REQUEST SIZE: 32768
mk_server/mk_http.c:323] mk_http_handler_read() [FD 128789] read 16
mk_server/mk_http.c:353] mk_http_handler_read() [FD 8] Retry total bytes: 16
mk_server/mk_http.c:1470] mk_http_sched_read() [FD 128789] HTTP_PARSER_PENDING
mk_server/mk_stream.c:88 ] mk_channel_write() [CH 8] CHANNEL_EMPTY
mk_server/mk_scheduler.c:566] mk_sched_event_read() [FD 8] Connection Handler / read
mk_server/mk_http.c:278] mk_http_handler_read() MAX REQUEST SIZE: 32768
mk_server/mk_http.c:323] mk_http_handler_read() [FD 128789] read 16
mk_server/mk_http.c:353] mk_http_handler_read() [FD 8] Retry total bytes: 16
mk_server/mk_http.c:1470] mk_http_sched_read() [FD 128789] HTTP_PARSER_PENDING
mk_server/mk_stream.c:88 ] mk_channel_write() [CH 8] CHANNEL_EMPTY
mk_server/mk_scheduler.c:566] mk_sched_event_read() [FD 8] Connection Handler / read
mk_server/mk_http.c:278] mk_http_handler_read() MAX REQUEST SIZE: 32768
mk_server/mk_http.c:323] mk_http_handler_read() [FD 128789] read 2
mk_server/mk_http.c:353] mk_http_handler_read() [FD 8] Retry total bytes: 2
mk_server/mk_http.c:1456] mk_http_sched_read() [FD 128789] HTTP_PARSER_OK
mk_server/mk_http.c:704] mk_http_init() [FD 8] HTTP Protocol Init, session 0x30fe50
mk_server/mk_http.c:842] mk_http_init() [FD 8] STAGE_30 returned -1
mk_server/mk_header.c:393] mk_header_set_http_status() Set HTTP status = 200
mk_server/mk_socket.c:40 ] mk_socket_set_cork_flag() Socket, set Cork Flag FD 8 to ON
mk_server/mk_http.c:223] mk_http_request_prepare() [FD 8] HTTP Init returning 0
mk_server/mk_stream.c:105] mk_channel_write() [CH 8] STREAM_IOV, wrote 171 bytes
____________________________________________________________________________
I dug into the output further and sorted the issue to one problem that I cannot understand, the mk_channel_write() [CH 8] CHANNEL_EMPTY line which comes in rtems, but doesn't on linux.
Secondly, I also observed that there is a warning Warning] Could not set TCP_FASTOPEN that comes on rtems. This is also a recent observation that needs to be investigated.
Alternative I am working on at the moment
The method used by monkey for inter-thread communication looks very similar to message queues. So presently I am working to re-write the whole event channel subsystem using message queues. As mentioned in the mail, I have started working on this yesterday afternoon, and hopefully will finish it by today afternoon/today night.
