Hi there,
We have seen a lot of unavailability on lichess this week. I'm facing a server configuration issues that gives me sleepless nights.
For those who care, I'm getting TCP SYN flood warnings in the kernel logs. I've tried many configurations tweaks using sysctl without much success: The site keeps getting down after around 1 hour of
activity, depending on the number of players online.
I'm now trying to tweak (read = hack) the java application server (netty) to increase its receive buffer stack size and backlog.
Until the right fix is found, we endure episodic failures, generally characterized by the infamous "Error 500" screen.
Sorry for the inconvenience, I hate it as much as you do. I'm working on it.
Cheers
just in case no one has said it today. Thank you for providing an awesome chess server. simple, easy to use and pleasing to the eye.
maybe an older working version of lichess can be swapped in for the non working one until a fix is found?
nope, because the problems only arise when there are lots of people playing. So I cannot investigate it if it's not online
Thanks raw03, some people said it before but it's always a joy to hear it ;)
You don't think part of the problem is all those little green dudes on the online page do you?
I notice those always took the longest amount of time to load...and they annoy me anyway. :D
after all it was one of the things that got broken during the rewrite but fixed later, before those 500s started popping up
Mephos: Yeah pretty sure. The problem is not bound to a single feature but to all of them, at a lower level.
lamobot: You bet I read it. Plus tons of other papers on TCP and kernel settings. Didn't make a difference.
Well it did make a difference. I don't have these kernel messages anymore. But the app still fails after a variable duration.
and by "fail" I mean take an infinite time to answer any request, without any warnings in logs