Introduction
Just adding my notes here in case it helps anyone else who finds themselves in a similar situation to me.I have two machines each with an Intel Xeon Phi Co-Processor board. This (in my basic view of the world) is like an extra machine inside the host machine but contains 240 processors clocking in at 1Ghz each. The co-processor appears to use an Intel specific Linux system called k1om (I think it may be derived from SuSE) so Ubuntu and CentOS specific commands do not apply here.
For the purposes of this guide the following are my assumptions:
- I will use the terms MIC, co-processor to refer to the co-processor board and the term host to refer to the system physically containing the co-processor board.
- Our LDAP server is running at 192.168.161.75 with an LDAP base of hpc.myorg.com
- hvan03 192.168.161.69 – this is the host machine containing the co-processor board
- hvan03-mic0 192.168.161.169 – this is the co-processor board
The following websites/files are helpful for configuration:
* Intel MPSS Download Page
* Cluster Setup Guide
* MPSS User Guide
* MPSS LDAP User Guide
MPSS Installation
The MPSS rpms are available from The MPSS Downloads Section. I placed the RPM files into my personal Yum Repository. These MPSS RPMs are installed onto the host (hvan03) via a Puppet configuration.
The MPSS service can be managed with:
systemctl stop mpss
systemctl start mpss
MIC Configuration
The mpss service should be stopped when making any config changes.
Upon initial installation of the MPSS software it is necessary to initialise the MIC configuration with:
micctrl --initdefaults
This creates the necessary config files under /etc/mpss for the MIC which can then be modified.
The current MIC configuration can be viewed at any time with:
micctrl --config
At this point if the MPSS service is started the co-processor will be accessible via SSH at the address specified in micctrl. e.g.
ssh root@172.31.1.1
The password for the root user on the MIC (hvan03-mic0) will be the same as the root password of the host (hvan03). The password hah is copied from the shadow file upon using –initdefaults.
Each MIC (co-processor board) is configured on the host (hvan03) at:
/etc/mpss/mic0.conf
This config file includes details of the co-processor hostname, IP address, root directory location, image to be used
The filesystem for each MIC is under:
/var/mpss/mic0
e.g. hvan03:/var/mpss/mic0/{etc,home,root,var}
Networking Configuration
It is necessary to create a bridge device on the host system (hvan03)
Create a bridge config file:
vi /etc/sysconfig/network-scripts/ifcfg-br0
With the following contents:
DEVICE=br0
TYPE=Bridge
ONBOOT=yes
DELAY=0
NM_CONTROLLED="no"
MTU=9000
BOOTPROTO=dhcp
NOZEROCONF=yes
Amend the primary network interface on the host (hvan03) for the bridge interface:
vi /etc/sysconfig/network-scripts/ifcfg-eno2
Add the following:
BRIDGE=br0
MTU=9000
The mic0.conf will contain a ‘Network class=’ line with default settings as generated by the micctrl –initdefaults command earlier. Amend the network settings of the co-processor :
vi /etc/mpss/mic0.conf
Change the ‘Network class’ line to the following:
Network class=StaticBridge bridge=br0 micip=192.168.161.169 modhost=yes modcard=yes
Starting the MPSS service at this point should make the MIC (hvan03-mic0) accessible via SSH at the address 192.168.161.169 using the same root password as used on the host (hvan03). Hopefully from within the MIC you should be able to ping other machines on the same network (via the bridge).
LDAP Configuration
- This configuration all takes place on the host (hvan03) rather than from within the MIC (hvan03-mic0)
- I first downloaded the file mpss-3.4.3.k1om.tar file from the MPSS download site.
- I extracted this tarball to /var/shared/k1om
- This provides MPSS/k1om specific RPMs that can be installed within the MIC OS
- Stop the mpss service: systemctl stop mpss
- Add the location of the RPMs (on the host not the co-processor):
micctrl --rpmdir='/var/shared/k1om'
- Set the LDAP details
micctrl --ldap=192.168.161.75 --base=hpc.myorg.com
- Restart the MPSS service
systemctl start mpss
- Access the MIC via SSH using an LDAP username and password combination:
ssh jonnymcc@hvan03-mic0.hpc.myorg.com
- From within the MIC amend the /etc/ssh/sshd_config file to restrict logins and allow all users to connect from only from the host (but the admin can also ssh from another box):
vi /etc/ssh/sshd_config
AllowUsers *@192.168.161.69 jonny@192.168.161.75
NFS File Share Configuration
We have to mount the shares directly as there are no autofs packages for k1om.
Create a mount point for the shared NFS file share:
mkdir /var/shared
Amend /etc/fstab
vi /etc/fstab
Add the following:
hpc-nas:/shared /var/shared nfs _netdev,rsize=32768,wsize=32768,timeo=14,hard,intr 0 0
hpc-nas:/ldap_user_homes /home nfs _netdev,rsize=32768,wsize=32768,timeo=14,hard,intr 0 0
Remount the fstab
mount -a
Software Installation
I made use of the RPMs extracted to /var/shared/k1om from the mpss-3.4.3.k1om.tar file to install software that might be useful to our group:
cd /var/shared/k1om
rpm -ivh libperl*.rpm
rpm -ivh perl-5*.rpm
rpm -ivh perl-modules-5.14.2-r7.k1om.rpm autoconf-2.68-r7.k1om.rpm gnu-config-20111111-r1.k1om.rpm
rpm -ivh rpm-build*.rpm
rpm -ivh autoconf-2.68-r7.k1om.rpm
rpm -ivh autoconf-doc-2.68-r7.k1om.rpm
rpm -ivh boost-1.49.0-r2.0.k1om.rpm
rpm -ivh boost-dbg-1.49.0-r2.0.k1om.rpm
rpm -ivh boost-dev-1.49.0-r2.0.k1om.rpm
rpm -ivh boost-staticdev-1.49.0-r2.0.k1om.rpm
rpm -ivh boost-test-1.49.0-r2.0.k1om.rpm
rpm -ivh cmake-2.8.7-r2.3.k1om.rpm
rpm -ivh cmake-dbg-2.8.7-r2.3.k1om.rpm
rpm -ivh cmake-doc-2.8.7-r2.3.k1om.rpm
rpm -ivh gccmakedep-1.0.2-r3.k1om.rpm
rpm -ivh gccmakedep-doc-1.0.2-r3.k1om.rpm
rpm -ivh gnu-config*.rpm
rpm -ivh libarchive2*.rpm
rpm -ivh liblzma5*.rpm
rpm -ivh libboost-*.rpm
rpm -ivh libgmp10-5.0.4-r0.k1om.rpm
rpm -ivh libgmp-dbg-5.0.4-r0.k1om.rpm
rpm -ivh libgmp-dev-5.0.4-r0.k1om.rpm
rpm -ivh libgmp-doc-5.0.4-r0.k1om.rpm
rpm -ivh libgmp-staticdev-5.0.4-r0.k1om.rpm
rpm -ivh libgmpxx4-5.0.4-r0.k1om.rpm
rpm -ivh libmpc2-0.8.2-r0.k1om.rpm
rpm -ivh libmpc-dbg-0.8.2-r0.k1om.rpm
rpm -ivh libmpc-dev-0.8.2-r0.k1om.rpm
rpm -ivh libmpc-doc-0.8.2-r0.k1om.rpm
rpm -ivh libmpc-staticdev-0.8.2-r0.k1om.rpm
rpm -ivh libmpfr4-3.1.0-r1.k1om.rpm
rpm -ivh libmpfr-dbg-3.1.0-r1.k1om.rpm
rpm -ivh libmpfr-dev-3.1.0-r1.k1om.rpm
rpm -ivh libmpfr-doc-3.1.0-r1.k1om.rpm
rpm -ivh libmpfr-staticdev-3.1.0-r1.k1om.rpm
rpm -ivh m4*.rpm
