Upgrading to xorg-server with USE=-hal appeared to make things run a tad faster. However, some really strange behavior with keypress events started to occur. I tried several different variants for keyboard layout, setting special keys etc but I still got stuff like “right ctrl is return” or “arrow down inserts a space and line down”. Adding to xorg.conf:
Section "InputClass" Identifier "keyboard-all" Driver "evdev" Option "XkbLayout" "se" Option "XkbVariant" ",qwerty" MatchIsKeyboard "on" EndSection
Made it work again. I was under the impression that you would have to configure less in xorg.conf after upgrading. But hey X works all dandy now.