All of the following issues occurred in a particular mixed GPFS 3.1 / GPFS 3.2 environment, utilizing multiple clusters with remote mounts.
All issues occurred prior to GPFS 3.2.1 PTF1-12, unless otherwise specified.
Problems that have not been verified as resolved remain listed for further testing and analysis.
Please note that issues where one or more specific APAR or defect numbers are not cited may exist as edge cases exclusive to the GPFS environment at Penn State.
- Single nodes can cause a cluster-wide deadlock, indicated by long waiters on one or more nodes.
(Deadlocks caused by token manager bugs have been resolved by IZ17622 and IY89771. Further deadlocks have occurred, up to and including 3.2 PTF1-12 in a non-mixed cluster. There are two outstanding PMRs to address this. Deadlock situations of this nature are expected to be remedied with the upgrade to 3.3).
- Extended network outages (hours) on one or more nodes have resulted in cluster-wide deadlock rather than clean failover.
(This is believed to be directly related to the issue noted above; however, staff in AIT have not been able to replicate the error in test environments and therefore cannot ascertain its cause or current state of risk. This is a known issue up to 3.1 PTF 25; AIT has upgraded to 3.2 PTF1-12. All documented cases of this have been addressed in GPFS 3.3).
- GPFS has demonstrated sensitivity to network irregularities that have typically gone unnoticed by most services.
This resulted in one of two situations:
- The node lost membership in a cluster, and all its filesystems were unmounted.
- The nodes were unable to complete an I/O operation within the timeout window, resulting in the expiration of the deadman switch timer and a kernel panic if said node has outstanding I/O requests.
- AIT has not experienced any security issues with the GPFS product to date.
Certain design characteristics have directly shaped AIT's implementation of GPFS. For further information, please send an inquiry to the IT Service Desk.
- Chmod and associated system calls delete all NFSv4 ACLs, and if *-k nfs4* is set, it utilizes a translation of the Mode Bits for effective ACLs.
(Penn State uses a customization added by IBM to ignore all chmod related calls as a temporary workaround. for the long-term, AIT would like chmod to alter the ACLs appropriately, in an agreed upon or customer-controlled fashion.)
- NFSv4 ACLs that are set by GPFS through its POSIX translation are brittle (they can easily end up with ineffective rules), and are incompatible with Windows/MSDN ACL rules [e.g., all
deny ACEs before all allow ACEs].
(Penn State would like to have an option to completely disable the use of the lower 9 Mode Bits, [e.g User:Group:Everyone] on a filesystem or define custom translation of said lower 9 Mode Bits to effective ACLs).
- Filesets are limited to 1,000 per filesystem. IBM has indicated that this is due to a lack of scalability for that feature, and would negatively impact performance if increased.
(Penn State needs this for proper home directory management; however, AIT is making use of a pseudo-"Directory Quota" feature added by IBM that utilizes group quotas.)
- Implementing more than 30,000 quotas per filesystem on GPFS versions prior to 3.3 results in severe performance degradation.
(Penn State uses 16 filesystems for user home directories, containing approximately 150,000 quotas).
- Some quota operations can cause an extended pause on a heavily loaded cluster (e.g. mmrepquota; mmdefedquota).
- Default group quotas do not work.
(This needs to be retested once all filesystems have been upgraded to 3.2).
- Quotas do not instantly update. in order to get accurate data, the mmlsquota -e switch ( a function not available in the API) has to be used to poll all nodes for updated info, and even then some count of blocks/inodes may be "in doubt."
- All quotas are enforced at the time of a write, not when a file is opened, but many applications only check for errors on open calls rather than write calls.
(While the current behavior is appropriate for a "hard quota," to truly prevent a user from abusing resources, a much more elegant way is to first prevent the user at a smaller "soft quota" limit using enforce on open).
Recently resolved / improved
- Prior to GPFS version 188.8.131.52, GPFS periodically responded to nfsd's MACRO call to the filesystem by attempting to delete the dentry, rather than return the appropriate value. This led to a NULL pointer dereference, causing a kernel panic.
- Prior to GPFS version 184.108.40.206, in Linux, the interpretation of groups fail after first 32. This has been raised to at least 1,024 groups.
- AIX is still limited to 128 groups (see also GPFS FAQ #5.7).
- Prior to GPFS version 220.127.116.11, inherited folder ACLs incorrectly display FileInherit ACEs (which are also not DirInherit) as having InheritOnly cleared, the opposite should be the case (see IZ21698).
NFS Gateway Issues
- NFSv4 spec provides for the capacity to report quota for a specified file path in GETATTR, but this has yet to prove itself as functional for Penn State's environment.
CIFS Gateway Issues
- Unusually high load seemed to trigger tdb_lock failures in Samba 3.0.30.
(This has not been witnessed since the number of servers was increased and the version was upgraded).
- Samba reports incorrect quota information.
(Penn State uses a script to report huge values to avoid incorrect "over quota" errors).
- Samba uses upper-case Kerberos realm names ("Domain" names) regardless of the case of the config files, causing failure with lower case realms.
(Penn State has modified the code to disable the toupper_ascii_fast and unix_strupper calls; overall impact of this change, outside of proper realm name handling, is currently unknown).
- Samba only checks for certain service principals (like hostname) and stops, rather than processes, the entire keytab.
(Penn State has patched the code to force a loop through all keytab entries, allowing for the use of DNS aliases as Kerberized service names).
- Changes to GPFS sharemode introduced with GPFS 3.2.1-12 for Linux cause problems for Samba.
(See this thread for more details and a workaround).
Mac OS X
- Starting with versions OS X v10.5, the Finder misinterprets mode permissions as authoritative over ACLs and falsely denies access. A problem report will be opened with Apple after further investigation.
- Kerberized NFS is not available prior to version 10.5.
- There is a resource fork issue with NFS from Mac.
- Mac extended file attributes may prevent files from being copied into NFS-mounted GPFS (still under investigation).
- Renewing tickets for Mac is a bigger problem for NFS due to immediate service interruption (CIFS only cares during connect).
- OS X only supports NFSv3, which has the following limitations:
- it does not support pseudofs, and thus requires an automounter or mount scripts to emulate said behavior.
- NFSv4 ACLs are not supported.
- The performance of the v4 client on Linux is dramatically slower than its equivalent on AIX, Solaris, or Linux v3 clients.
- Until relatively recent releases, the rpcgssd required Kerberos service principal and keytab for the client system.
- The rpc.gssd has insisted on "nfs" service principals for clients rather than "host" (as of version 1.1 nfs-utils allow "root" and "host" principals).
- NFSv4 replication (server redundancy) is currently unsupported.
- Tickets are not automatically renewed on active Linux clients. (k5start is being used as a workaround).
- mount.cifs kerberos support is not available since post-mount.smb code fork, though it is listed as a future TODO (see http://linux-cifs.samba.org/).
- The special ".." link in the top level of an NFS volume will occasionally fail to properly map to the parent folder of the client's mount point. (This is temporarily being solved with a remount, but it is still under investigation).
- There are sporadic file execution issues; when running programs stored over NFS-mounted GPFS, one may receive any of the following errors:
- "Cannot open or remove a file containing a running program."
- "bad interpreter: Cannot open or remove a file containing a running program." (for interpreted languages)
- "A file or directory in the path name does not exist."
- "Text file busy."
- "0403-015 Cannot access a required executable file. It is in use."
- "Stack Overflow"
- "pwd: There is an input or output error."
- (Partial dynamic library load issues)
exec(): 0509-036 Cannot load program /usr/local/bin/svn because of the following errors: 0509-150 Dependent module /opt/subversion/lib/libsvn_client-1.so could not be loaded. 0509-022 Cannot load module /opt/subversion/lib/libsvn_client-1.so. 0509-026 System error: A file, file system or message queue is no longer available.
- There are stat() failures on objects that are read from a directory list.
- Likely related to the stat and execution issues, errors like the following are ocassionally encountered:
- ls: .: The file access permissions do not allow the specified action.
- ls: 0653-341 The file /usr/local/bin/screen-4.0.3 does not exist.
- AIX has an issue where portmapper will, from time to time, hang machines.
(AIT is not certain if this exact issue or other related issues have been experienced in the Penn State environment: IZ52039)
- The ls command fails to work over NFSv4, though other commands making use of the stat() API are ok. (This is currently under investigation).