Andrew Cooke: C[omp]ute

SuperMicro X10SDV-TLN4F/F with Opensuse Leap 42.1

From: andrew cooke <andrew@...>

Date: Sat, 26 Mar 2016 16:57:36 -0300

New computer :o)

I'm typing this on one of these -
http://www.supermicro.com/products/motherboard/Xeon/D/X10SDV-TLN4F.cfm -
that's 8 Xeon cores (16 hyperthreads) with 64GB buffered ECC memory
(Samsung 4x16 DDR4).

The graphics card I'm using is an ASUS NVidia GTX 960
https://www.asus.com/Graphics-Cards/GTX960-MOC-4GD5/specifications/ (it's
small enough to fit the enclosure, but still reasonably powerful for
GPGPU work).

Disks are 2TB RAID spinning and 220GB SSD.  Keyboard is an old IBM
spacesaver, with PS2 to USB.

Operating system is Opensuse Leap 42.1, connected to my local ISP using
PPoE(!).

Enclosure is an old Silverstone case, about the size of a large shoebox.
It's pretty quiet, but you can hear the GPU fan, slightly.

Everything works very smoothly, but it took some time to get here.  Below
I'll detail the main issues to help others on the way.


First hurdle was getting to the BIOS setup screen.  It's slooooow to
boot and I may simply have run out of patience, or perhaps it needs
memory installed.  But anyway, you seem to have just 1 second in which
to hit DEL when the correct screen comes up.

So once you get into the setup, disable the "quiet" boot (gets rid of
the pretty Supermicro logo, but doesn't display any more until later in
the boot - at the start you get just a single line ending with cryptic
hex digits) and set the pause time to 15s or so.


Next issue was booting from disk.  I think there were two separate fixes,
but it could have been just one that addressed the real problem.  Anyway,
things work for me with UEFI disabled (legacy boot) and also with all the
alternate boot options disabled (this is hard to describe, but on the
BIOS setup boot screen, below the usual list of devices, there are
further entries - select each and disable that fucker).


Third issue was USB3 to my backup disk.  The board has just two USB3
ports on the rear panel, and it seems like the cheapo PS2 keyboard
converter was dragging things down to USB2(?) so I moved the keyboard to
a USB2 on the case and forced USB3 in BIOS (disabled legacy).  That got
USB3 to the disk.


Fourth issue was getting Nvidia to start.  I did my initial install using
the VGA port on the mobo.  Connecting the graphics card didn't change
anything (except the errors in the xorg log) - the chip on this card is
too new to be supported by Nouveau (this is with two monitors - one
connected to VGA and one to the graphics card).  So you need to install
the proprietary drivers.  That's easy in OpenSuse (add the repo and run
"zypper inr" as described at https://en.opensuse.org/SDB:NVIDIA_drivers),
but still doesn't work.  What finally got things working was moving the
jumper JPG1 on the mobo - you can't just remove this, you actually have
to shift it to pins 2+3, which will disable the VGA hardware and allow
Nvidia to come up cleanly (I don't understand why the two can;t coexist,
but there you go).


That's all, so far.  For the 24 or so hours it's been up, it's been
completely solid.  No idea what the total power draw is, but the (old,
Silverstone) PSU is rated at 450W.

I guess I should go spin up a few VMs running Oracle and giggle.

Andrew

Fan and USB issues

From: andrew cooke <andrew@...>

Date: Fri, 15 Apr 2016 18:46:32 -0300

A couple of weeks in and a bit more experience.


I was surprised to find that sometimes (no idea the cause or what
"fixed" it) the board would start regularly "surging" the fans -
spinning them up to maximum for a few seconds every 20s or so.  This
was extremely annoying (although I don't think there were any terrible
consequences for the hardware) and hard to ignore.

I contacted Server Factory (the people I bought the board from) and
they said this was usually fixed by a BIOS update.  So, when I had
enough free time to do it carefully, I updated the BIOS (via a
bootable USB - http://www.chtaube.eu/computers/freedos/bootable-usb/).

However, that didn't help, so I contacted Server Factory again, who
asked for details of my chassis and fans.  Up to that point I'd
blindly assumed this was a BIOS issue and not really considered my
hardware, but that email pushed me to check things.  And it was soon
clear that the surging only occurred when my case fan was connected to
the mobo (the case is a Silverstone Sugo 5).

Thinking some more, I guessed that the mobo was sensing the fan RPM
using the fourth pin and, if the pin was not spinning fast enough,
kicking everything to max RPM.  So I suspected that my (old) fan was
sluggish at low voltages.  And buying a new fan (PWM with 4 pins)
seems to confirm that - I have had no surging so far with the new fan
installed.


The other surprising behaviour I've had has been detection of USB
devices.  I have a rather odd / old collection of devices, which are
currently arranged as follows: on the rear USB3 port, I have
Thermaltake disk caddy; on the front USB2 port I have a powered hub
that chains out to a CD reader, an audio device, and my monitor;
chained from the monitor I have my keyboard via a PS2/USB adapter.

Given that setup, I sometimes have to boot multiple times, or
reconnect devices, before everything is stable.  Problems include the
keyboard not working (even though the integrated pointer nub does) and
the disk caddy not making a high speed connection.  Also, the powered
hub appears to supply power to the case fan and mobo LEDs even when
everything else is powered off.


In both these cases part of the blame is likely with me / my
hardware.  But there's also some questions about the board - I'd love
to hear if anyone else has had similar experiences.

I should close by adding that, apart from these issues, it has
continued to be solid.  I've had all cores running with no issues, and
had multiple large VMs happily rolling along.  Would buy again.

Andrew


PS One more thing.  This generation of Intel chipsets don't play well
with old Sandforce SSD chipsets.  So I can't mount an old OCZ SSD I
have lying around.

A Few Weeks On...

From: andrew cooke <andrew@...>

Date: Fri, 6 May 2016 17:47:18 -0300

Apart from one unexplained crash (which was while I was watching a
video and suspect was flash / nvidia driver related) this has been
extremely solid.

Having a huge (for me at least) amount of memory means I can have
multiple VMs open, and switch between projects quickly (my work often
involves maintaining software that interacts with Oracle, web
services, etc, so I organise it around test VMs that are configured
for the appropriate environment).

The IO handling also seems to be a big improvement over what I had
before - my spinning disk RAID seems to be much faster.  I guess this
is partly just caching in memory, but may also be improved IO
management.  I never seem to get a slow machine just because disks are
being thrashed.  In fact I cannot tell - from general machine
performance - how much the disks are being used.

The only drawback is that - as expected - single thread speed is not
much better than what I was used to (on a fairly old CPU).  But the
most CPU intensive process I know - building julia - can actually be
spread acorss multiple cores (make -j 16) and so is significantly
faster.

In short - I am really happy with this board / machine.

Andrew

Comment on this post

C[omp]ute

Personal Projects

Last 100 entries

SuperMicro X10SDV-TLN4F/F with Opensuse Leap 42.1

Fan and USB issues

A Few Weeks On...