A convenient framework for toString()

It is a not only praxis, but pest practice, to override toString() in new classes. This article presents a convenient framework for preserving the look and feel of toString() outputs.

  

formatArg() creates a Map from it’s parameters, a map that contains a single entry, keyed by the name of the value to display. newHashMap is a type-safe Map creation method:

formatObjectToString() returns a String given a List of Maps (which Maps can be created by formatArg()). This version has hard-coded formatting, but it would be simplicity itself to add a Pattern by which the formatting could be dynamically controlled.

By way of example, class BasicNumericDataCollectorAction from the DIrectory Server Log Analyzer uses formatArg() and formatObjectToString() in its overridden toString() method by presenting in String form the statistical items that are calculated by the class:

Links:

1. convenience framework
2. usage example

Paste from clipboard – emacs

Having trouble with pasting into Emacs? Use the function ‘clipboard-yank’, thus:

(global-set-key "\C-y" 'clipboard-yank)

I win as Black against the Colle System

The Colle System is a very solid, but rather crampy system used by White. It one of the systems in the Queen’s Pawn game. In this game, I am playing Black against a Facebook opponent: Black takes advantage of some neat tactical points, and finishes up with a nice checkmate.

Facebook opponent – Gardner, October 2009

1. d4 d5 2. e3 Nf6 3. Nf3 e6 4. c3 Bd6 5. Bb5+ Nbd7 6. O-O a6 7. Bd3 e5

8. a3?

White goes wrong here. In the opening, there is no time to waste on such a pawn move. The Black dark-squared Bishop is already restricted in its movement since it cannot move to c5 or b4 because those squares are protected by White pawns, so there is no point in over-protecting b4 with 8. a3. Black is threatening to play … e4, which would fork the White Knight and Bishop.

The correct move is 8. dxe5 followed by 8. … Nxd5 9. Nxd5 Bxd5 10. Nd2.

8. … e4

Black responds by pushing the e-pawn and forking the White Knight and White light-square Bishop. White can save only one of these pieces.

9. Ne5

Generally speaking, White should have saved the Bishop instead of the Knight.

9. … exd3

Black now has the two Bishops, which can be an advantage, but of course it depends on the position.

10. Nxd7?

Again, generally speaking, the side which is down in material should not trade pieces. White was already down one piece, by trading the Knight White improves his opponent’s position by eliminating material when Black is already ahead in material, and, after Black captures the Knight on d7 with his Bishop, Black will enjoy a lead in development, that is, Black will have three developed pieces, while White has zero developed pieces. With a lead in development and one piece ahead, White has objectively lost the game. If Black plays accurately from now on, White has no chance at all, and this proves to be the case.

10. … Bxd7

The diagram above shows the board with Black’s tremendous lead in development. To be sure, Black’s far-flung pawn on d3 is toast, but that is a small price to pay for such a lead in development and a material advantage.

11. Re1

There was no point in moving that Rook yet. Best to wait until the last minute before moving the Rooks (generally speaking). Black has two pieces aimed at White’s Kingside already (the Bishop on d7 and its partner prelate on d6): by placing his Knight on g4, Black will be attacking h3 twice, and h3 is defended by only one piece – the White King.

11. … Ng4 12. g3

White’s best move of the game! This blocks the Black Bishop on d6 and removes the double attack on h3. Trouble is, it’s too late. Black’s superior position is getting better with every move, and White still has not developed another piece.

12. … Qf6?!

Black goes astray. The correct move was 12. … Nxh2! If White takes the Knight with 13. Kh2, it is all over for White very quickly: 13. … Qh4+ 14. Kg2 Qh3+ 15. Kg1 Bxg3! 16. fxg3 Qxg3+ 17. Kh1 Bg4 18. Qd2? Bf3+ 19. Qg2 Qxg2 checkmate. Of course, White could sacrifice the Queen with 18. Qxg4 Qxg4 19. Kh2 0-0-0 20. e4 Rd6 from here, it’s mate in 6 moves.

13. Rf1 (arranging two defenders for f2, which was attacked twice by Queen and Knight) O-O 14. b4?

Once again, White does not develop a piece. Black has a crushing advantage: everything except the two Black Rooks are pointed at the White King, in spite of this, White is wasting time moving pawns.

14. … Qh6

Black is overly aggressive here, one of the Rooks should have been placed on e8.

15. Kg2??

White seems depressed, he walks right into checkmate. When playing chess and getting hammered as White is in this game, it is possible to lose objectivity and play badly. Black has delivered blow after blow for several moves, White has been on the defensive for about 7 moves and now crumbles completely under that pressure. The following checkmate sequence is shown move for move:

15. … Qxh2+

First, force the White King out of his bolt-hole (this is possible because of the g3 move).

16. Kf3

16. … Nxf2

Next, restrict the White King’s movements. This is akin to forcing him from his castle onto the field of battle and then besieging him from many sides at once.

17. Qxd3

Note that White is so restricted in his movements, practically any move he makes is followed by checkmate. Now, the coup de grace:

17. … Bg4 checkmate

Note that at the end of the game, White’s dark-square Bishop, his Queen’s Knight, and his Queen’s Rook have never moved from their original squares. A seemingly minor mistake like a pawn move leads to disaster for White.

Safari Bookmarks sorting

Outside of Windows itself, the biggest software design error of all time is the missing “Sort Bookmarks” feature in Safari. On the Apple forums, users blithely suggest “clicking and dragging”, “create folder in Finder or Path Finder, move bookmarks to folder which sorts them move them back to Safari”, “export to Firefox and re-import”, and other arcane, 1970s-era solutions as if these were really decent solutions. Sure, they all work, but pul-lease. The “new folder in Finder” is almost funny in its torturousness. It’s like siphoning your gas into into the filling station tanks and then re-filling your tank when you need petrol.

There are a number of third-party solutions to this problem, thankfully, but I just wonder how it is that software can be designed without something as simple as sorting pieces of information. Honest to God, I think this is Apple’s little joke on the rest of us: “watch ‘em suffer for a while, we’ll put it in later”.

pestilential browser specific code

One of the things that is most annoying to surfers of any level of experience is the proliferation of pestilential browser code, code that adheres to the DNA of a specific browser, often because of idiotic Javascript (which should be called ECMAScript, or WasteMyWholeDayOnPseudoLanguagesScript, because it certainly has nothing to do with the elegance that is Java).

For example:

This little gem showed up after 4 pages visited on the website, it may be just me, but one would think that one would know after four pages that the visitor was not using Firefox (was Safari, I go back and forth between Firefox and Safari, depending on my mood, mostly: Safari is more elegant and has superior visual rendering, whereas Firefox is seemingly infinitely extensible).

The problem is simple and straightforward: Jxxxscript makes programmers lazy. Its unstructured nature is well-suited for some fanboy sophomorish programming project, but certainly not for users in the wild where standardization and simple look and feel are mission-critical. If you’re writing something that only works on a specific browser, then you do not understand your requirements, and possibly may be cursed with an inadequate (and embarrassing) test harness.

When presentations fail …

Presentations, often accompanied by a “deck” or “slide deck”, are given to prospective clients, customers, and those with which the presenter desires an advantageous relationship. Presentations are ineffective when:

• they are not targeted to a specific relationship (scatter-shot approach), If you aren’t targeting a specific relationship, you’re wasting everybody’s time
• the present solutions for targets that are not articulated (“My company’s Exact Docutron corrects spelling at the rate 10,000 words per minute” – so what? Has the presentee indicated that the company’s clerks mis-spell at that rate?)
• they foolishly attempt to use the same old tired rationales. Try inventing something new for a change. The thought process of re-using material is a vain and pointless attempt to emulate object-oriented software reusability, but people don’t react like software, dude
• too long. If two or three needs cannot be articulated (or created) and solved in two or three slides, it’s too long. A presentation is not a time-share presentation where one gets a prize at the end automatically

If you are expecting your potential relationship to take the time to create one end of the relationship, you should take the time to create brand new material, or at least mostly brand new material. Think of it as a first date, and then think of those ridiculous soirees when a woman meets 30 men in 30 minutes. How effective is that except to the most superficial broad? Create something new, intelligent, pointed, focused … and short.

13,800,000 entry Sun Directory Server 5.2 patch 6 Benchmark yields 17,000 searches per second

Introduction

The Sun Directory Server provides the most scalable, high-performance LDAP data store for identity information in the industry and serves as the foundation for the new generation of e-business applications and Web services.

Sun Directory Server provides a high-performance, scalable LDAP and DSML directory services environment that supports multi-master relication for high-availability and redundancy.

The benchmark described in this post involved a requirement for a Consumer Directory Server for 13,800,000 user entries with Directory Server 5.2 patch 6 running on Sun Netra x4250 hardware and Solaris 10 update 7. The performance of Directory Server in a mixed LDAP operation environment is a mission-critical factor in proper authentication of user handsets in a large telecommunications company. The certification numbers to obtain were:

• 8,000 searches per second with simultaneous updates
• maximum 800 milliseconds for any single search
• minimum 70% CPU utilization (usr+sys)

Highlights of the benchmark:

• Sun Directory Server 5.2 patch 6 was installed on a single consumer node: a Sun Netra x4250, dual-CPU, 8-core (2.13 Ghz), 64 GB RAM running Solaris 10 update 7 using internal disk drives
• 13,800,000 entries loaded into the Directory Server database in 2 hours
• 17,000 searches per second sustained over 8 hours with no single response time in excess of 2 milliseconds, an average latency of less than 2 milliseconds with no update (replication) traffic
• 11,000 searches per second sustained over 8 hours with no single response time in excess of 2 milliseconds, an average latency of than 2 milliseconds in the presence of update (replication) traffic
• SLAMD was used to generate and report on load

Hardware Configuration

Consumer Server Node Hardware Specifications

• 2 quad-core 2.13Ghz Intel Xeon processors, a total of 8 cores
• 64 GB RAM
• Solaris 10 update 7
• gigabit ethernet

Consumer Server Node Storage Specifications

• ZFS used for Directory Server database files
• UFS used for Directory Server logs
• Seagate 10K 300GB SAS drives
• See Filesystem Cache Optimization Strategies by Brad Diggs

Solaris Kernel Parameters (except ZFS)

• autoup = 300
• TCP/IP parameters as set by nddconfig stock distribution

ZFS Configuration

• zfs:zfs_arc_max = 0xA80000000 (42 GB )
• zfs:zfs_prefetch_disable = 1
• zfs:zfs_vdev_cache_bshift = 13
• zfs:zfs_cache_size = 0
• 8k record size

UFS Configuration

• ufs:freebehind = 0
• ufs:smallfile = 2147483647
• mounted with forcedirectio

Test Client Hardware Specification

• 2 quad-core 2.13Ghz Intel Xeon processors, a total of 8 cores
• 64 GB RAM
• Solaris 10 update 7
• Java 1.6.0_16
• gigabit ethernet

Directory Server and Database Configuration

Directory Server Software

Sun Directory Server 5.2 patch 6 was used for this benchmark. All tuning parameters were left at default values with exceptions listed in the Tuning section below. The slab allocator libumem.so was used instead of the default single-threaded memory allocation library.

Test Load Generation and Reporting

SLAMD Version 2006 was used for LDAP load generation, data collection, and reporting. SLAMD is an open-source distributed load generator originally developed by Sun Microsystems. The principal developers of SLAMD are Neil A. Wilson and Terry J. Gardner. SLAMD consists of the following components:

• server software running in a Tomcat servlet container
• Five (5) stand-alone Java programs (“SLAMD Clients”) for load generation
• stand-alone Java programs (“SLAMD Resource Monitor Clients”) for collecting operating system metrics

Tuning

Sun Directory Server patch 6 Configuration

• number of startup threads = 32
• database cache size = 1 GB
• entry cache size = 1 GB
• import cache size = 2 GB
• database mapping files in swapfs (/tmp) filesystem
• audit logging activated (records all updates)
• access log configured to log microsecond granularity
• access, error, and audit file rotation at 20 MB

Directory Server 5.2 patch 6 is a 32-bit process on Solaris Intel systems, therefore it is critical to keep the process size of DIrectory Server below 3.7 GB. To this end, the small database cache and entry cache sizes were used. The main caching entity was the ZFS cache, therefore the ZFS cache is primed at the beginning of test run, which prevents a test run from beginning with a “poisoned” cache. Observed process size after 8 hours of sustained load was 2503 GB. Note: the import cache is not used except during data bulk loads.

Bulk Load of Data into Directory Server

Data Characteristics

The data used for the database was taken from production customer Directory Servers, resulting in a very life-like benchmark. Each entry is of similar construction and is approximately 2302 bytes in size, including operational data and replication meta-data.

Preparing for Bulk Load

User data was extracted from customer production system backups and loaded into a Supplier (master) Directory Server. This Directory Server Supplier was used to generate replication traffic, but not otherwise part of the Benchmark. Once data was loaded into the Supplier it was tested using db_stat, then extracted using db2ldif -r to preserve replication data. The output of db2ldif -r was then imported into the Consumer Directory Server and replication between the Supplier and Consumer was enabled.

Bulk Load Execution

The user data, that is, the output of db2ldif -r, was loaded into the Consumer Directory Server using the ldif2db command. This process took approximately 2 hours.

After the database loaded into the Consumer Directory Server, the database, including the necessary index files was observed to approximately 15 GB.

Benchmark Scenarios

Test Clients

Five (5) SLAMD clients (stand-alone Java clients), with two (2) threads each generating search load from search filters constructed from actual customer data.

Test Scenarios

The Benchmark consisted of the following sequence, which was repeated in order for each benchmark run to ensure a level playing field for each sequence:

• The ZFS cache was primed with data from the database using the dd command

• a database cache priming job consisting of a 1200 second (20 minute) SLAMD job comprised of a series of LDAP searches using search filters derived from customer data
• a 5 minute null job – no activity, allows Directory Server to quiesce
• two simultaneous jobs: 1) a job generating ADDs, MODs, and DELs in the precise mixture derived from customer provided access logs from production systems. This job is schedule for 8 hours duration, but in reality takes longer because ADDed entries are removed at job completion. 2) a job executing searches using search filters derived from customer provided data.

Benchmark Results

The Consumer Directory Server sustained 11,000 searches per second and replication (ADD, MOD, DEL) simultaneously over an 8 hour period.

Conclusion

The certification search throughput was exceeded by over 3,000 searches per second, and response times were less than 1/800 of maximum tolerable response times.

The combination of Sun Directory Server 5.2 patch 6, Solaris 10 update 7, dual-cpu 8-core Sun x4250, and libumem.so makes a very fast, memory efficient Directory Server platform. The customer will deploy 8 identically configured servers, resulting in a maximum throughput of 88,000 searches per second simultaneously with updates.

BlogCatalog

Changes to Delta’s web site

Delta Air Lines recently made some changes to their web-site, for example, asking for email addresses and surname at login time. This is as a result of the merger of Sky Miles with Northwest’s frequent flyer program. Our hope had been that Northwest’s 1980’s vintage code would have been improved by Delta’s 1990’s vintage code, but the opposite has now occurred. Northwest has infected Delta’s Sky Miles system such that:

• incessantly asks for emails
• fails to report earned Medallion qualification miles as soon as the flight lands (takes 24 hours now)

The above are reminiscent of Northwest’s 1980’s vintage code, and while a small thing, is a step backwards for Sky Miles. Computer systems and IT systems should be moving forward, not backward. Trivialities are excellent predictors of future corporate actions and health, and a pointer to what goes on in the present. Delta’s powerful Sky Miles program interface has now become a 1980’s clunky software system, which is piss-poor now matter how you look at it.

How it will be spun: “we’ve competed our merger of World Perks and Sky Miles! Now our guests have all their miles IN ONE PLACE! Our customers have not lost a SINGLE MILE! This quick turn-around is indicative of Delta’s commitment to blah, blah, blah” (while true, it’s irrelevant, and is the very nature of spin, that is, find a positive aspect of a shitty thing and trumpet it)

The reality: “we chose the path of least resistance so that we could spin, and so we could reduce time-to-market, not the hard road of correctness and robustness in software, therefore making us better than the others in the long run. Also, we think our customers are so internet-unsavvy they will not notice our clunked-up, 1980’s interface to Sky Miles and its operational characteristics.” Dudes, reducing time-to-market only works when the product is better than the competition, not worse or equal (Microsoft is a perfect example: Windows has been riddled with software failures for years, is the most insecure operating environment ever invented, often crashes, and is generally a laughing stock, but by God, it’s the first).

Undoubtedly, some readers of this blog are thinking: “what does it matter? talk about trivialities!” It matters because:

• everyone, corporations included, should strive for the very highest and best at all times
• software and systems should always get better, not worse
• the hard decision road is almost always the best road
• spin of any form should be critically examined and debunked whenever and wherever possible, even if it’s harmless spin. For example, Ballmer’s most recent labeling of “other” browsers as “rounding errors” which while true from a numbers perspective is certainly not true from a growth and quality perspective. An important function of executives is to spin, and to provide direction for other spinners in the corporation, to be sure. It is the consumer’s job to critically examine the spin for flaws, and there will be flaws because the flaws are the reason for the spin in the first place.

Travel Day, 02-Oct-2009

Travel Day

Today is travel day, that is, the day I must interrupt my work and travel home. In this instance, Newark’s Liberty Airport to the Delta Fortress. Delta Air Lines will be operating the flight with a B73. After an uneventful flight – which is just like I like my flights – arrived early in Atlanta.

A B75 attended by support craft and food tenders at the Delta Fortress:

DS52p6 Iterations on x4600

Ran an SLAMD optimizing job using libmtmalloc on the 4600 with 8 cores, 32GB RAM, database on ZFS, achieved greater than 20,000 searches per second, using Sun Directory Server 5.2 p6. libmtmalloc is a multi-threaded implementation of the standard memory allocation library, see also malloc(), calloc(), free().