13,800,000 entry Sun Directory Server 5.2 patch 6 Benchmark yields 17,000 searches per second

Introduction

At one time, the Sun Directory Server provided the most scalable, high-performance LDAP data store for identity information in the industry and serves as the foundation for the new generation of e-business applications and Web services.

Sun Directory Server provides a high-performance, scalable LDAP and DSML directory services environment that supports multi-master relication for high-availability and redundancy.

The benchmark described in this post involved a requirement for a Consumer Directory Server for 13,800,000 user entries with Directory Server 5.2 patch 6 running on Sun Netra x4250 hardware and Solaris 10 update 7. The performance of Directory Server in a mixed LDAP operation environment is a mission-critical factor in proper authentication of user handsets in a large telecommunications company. The certification numbers to obtain were:

8,000 searches per second with simultaneous updates
maximum 800 milliseconds for any single search
minimum 70% CPU utilization (usr+sys)

Highlights of the benchmark:

Sun Directory Server 5.2 patch 6 was installed on a single consumer node: a Sun Netra x4250, dual-CPU, 8-core (2.13 Ghz), 64 GB RAM running Solaris 10 update 7 using internal disk drives
13,800,000 entries loaded into the Directory Server database in 2 hours
17,000 searches per second sustained over 8 hours with no single response time in excess of 2 milliseconds, an average latency of less than 2 milliseconds with no update (replication) traffic
11,000 searches per second sustained over 8 hours with no single response time in excess of 2 milliseconds, an average latency of than 2 milliseconds in the presence of update (replication) traffic
SLAMD was used to generate and report on load

Hardware Configuration

Consumer Server Node Hardware Specifications

2 quad-core 2.13Ghz Intel Xeon processors, a total of 8 cores
64 GB RAM
Solaris 10 update 7
gigabit ethernet

Consumer Server Node Storage Specifications

ZFS used for Directory Server database files
UFS used for Directory Server logs
Seagate 10K 300GB SAS drives
See Filesystem Cache Optimization Strategies by Brad Diggs

Solaris Kernel Parameters (except ZFS)

autoup = 300
TCP/IP parameters as set by nddconfig stock distribution

ZFS Configuration

zfs:zfs_arc_max = 0xA80000000 (42 GB )
zfs:zfs_prefetch_disable = 1
zfs:zfs_vdev_cache_bshift = 13
zfs:zfs_cache_size = 0
8k record size

UFS Configuration

ufs:freebehind = 0
ufs:smallfile = 2147483647
mounted with forcedirectio

Test Client Hardware Specification

2 quad-core 2.13Ghz Intel Xeon processors, a total of 8 cores
64 GB RAM
Solaris 10 update 7
Java 1.6.0_16
gigabit ethernet

Directory Server and Database Configuration

Directory Server Software

Sun Directory Server 5.2 patch 6 was used for this benchmark. All tuning parameters were left at default values with exceptions listed in the Tuning section below. The slab allocator libumem.so was used instead of the default single-threaded memory allocation library.

Test Load Generation and Reporting

SLAMD Version 2006 was used for LDAP load generation, data collection, and reporting. SLAMD is an open-source distributed load generator originally developed by Sun Microsystems. The principal developers of SLAMD are Neil A. Wilson and Terry J. Gardner. SLAMD consists of the following components:

server software running in a Tomcat servlet container
Five (5) stand-alone Java programs (“SLAMD Clients”) for load generation
stand-alone Java programs (“SLAMD Resource Monitor Clients”) for collecting operating system metrics

Tuning

Sun Directory Server patch 6 Configuration

number of startup threads = 32
database cache size = 1 GB
entry cache size = 1 GB
import cache size = 2 GB
database mapping files in swapfs (/tmp) filesystem
audit logging activated (records all updates)
access log configured to log microsecond granularity
access, error, and audit file rotation at 20 MB

Directory Server 5.2 patch 6 is a 32-bit process on Solaris Intel systems, therefore it is critical to keep the process size of DIrectory Server below 3.7 GB. To this end, the small database cache and entry cache sizes were used. The main caching entity was the ZFS cache, therefore the ZFS cache is primed at the beginning of test run, which prevents a test run from beginning with a “poisoned” cache. Observed process size after 8 hours of sustained load was 2503 GB. Note: the import cache is not used except during data bulk loads.

Bulk Load of Data into Directory Server

Data Characteristics

The data used for the database was taken from production customer Directory Servers, resulting in a very life-like benchmark. Each entry is of similar construction and is approximately 2302 bytes in size, including operational data and replication meta-data.

Preparing for Bulk Load

User data was extracted from customer production system backups and loaded into a Supplier (master) Directory Server. This Directory Server Supplier was used to generate replication traffic, but not otherwise part of the Benchmark. Once data was loaded into the Supplier it was tested using db_stat, then extracted using db2ldif -r to preserve replication data. The output of db2ldif -r was then imported into the Consumer Directory Server and replication between the Supplier and Consumer was enabled.

Bulk Load Execution

The user data, that is, the output of db2ldif -r, was loaded into the Consumer Directory Server using the ldif2db command. This process took approximately 2 hours.

After the database loaded into the Consumer Directory Server, the database, including the necessary index files was observed to approximately 15 GB.

Benchmark Scenarios

Test Clients

Five (5) SLAMD clients (stand-alone Java clients), with two (2) threads each generating search load from search filters constructed from actual customer data.

Test Scenarios

The Benchmark consisted of the following sequence, which was repeated in order for each benchmark run to ensure a level playing field for each sequence:

The ZFS cache was primed with data from the database using the dd command

a database cache priming job consisting of a 1200 second (20 minute) SLAMD job comprised of a series of LDAP searches using search filters derived from customer data
a 5 minute null job – no activity, allows Directory Server to quiesce
two simultaneous jobs: 1) a job generating ADDs, MODs, and DELs in the precise mixture derived from customer provided access logs from production systems. This job is schedule for 8 hours duration, but in reality takes longer because ADDed entries are removed at job completion. 2) a job executing searches using search filters derived from customer provided data.

Benchmark Results

The Consumer Directory Server sustained 11,000 searches per second and replication (ADD, MOD, DEL) simultaneously over an 8 hour period.

Conclusion

The certification search throughput was exceeded by over 3,000 searches per second, and response times were less than 1/800 of maximum tolerable response times.

The combination of Sun Directory Server 5.2 patch 6, Solaris 10 update 7, dual-cpu 8-core Sun x4250, and libumem.so makes a very fast, memory efficient Directory Server platform. The customer will deploy 8 identically configured servers, resulting in a maximum throughput of 88,000 searches per second simultaneously with updates.