Category Archives: Programming

Memory access patterns in high-level languages

Like many developers that work in high-level languages, I think don’t spend a lot of time thinking about memory access patterns. This is probably a good thing… for the most part, worrying about this is premature optimization. But there are times when it matters, and the compiler won’t magically “optimize” it away for you, even if you have optimizations turned on:

Code

Results

I tried this out after seeing this discussion thread on Quora this morning.

redis: Sorted Sets as iterable key-value work queues

I’m in the process of building an internal search engine at work, and first on the block are our network share drives. During a typical week, we have about 46,000 unique documents that are written to in our Lexington office. This number only represents things like Word docs, Excel spreadsheets, PowerPoint presentations, PDFs, and so forth. (A count of all touches is 3-4x higher.)

This presents some challenges: crawling the network shares at regular intervals is slow, inefficient, and at any given time, a large portion of the search index may be out of date, which is bad for the user experience. So I decided an incremental approach would be better: if I re-index each document as it’s touched, it eliminates unnecessary network and disk IO, and the search index is updated in close to realtime.

Redis

My second-level cache/work queue is a redis instance that I interact with using Booksleeve. The keys are paths that have changed, and the values are serialized message objects stored as byte arrays that contain information about the change. The key-value queue structure is important, because the only write operation that matters is the one that happened most recently. (Why try to index a document if it gets deleted a moment later?)

This is great in theory, but somewhere along the way, I naively assumed it was possible to iterate over redis keys… but you can’t. Not easily or efficiently, anyway. (Using keys in a production environment is dangerous, and should be avoided.)

Faking iteration

The solution was relatively simple, however, and like all problems in software development, it was solved by adding another level of indirection: using the redis Sorted Set data type.

For most use cases, the main feature that differentiates a Set from a Sorted Set is the notion of a score. But with a Sorted Set, you can also return a specific number of elements from the set. In my case, each element returned is the key to a key-value pair representing some work to be done.

Implementing this is as easy as writing the path to the Sorted Set at the same time as the key-value work item is added, which can be done transactionally:

Downstream, my consumer fills its L1 cache by reading n elements from the Sorted Set:

And there we have fake key “iteration” in redis.

A little syntactic sugar to have reference semantics for value types

In C#, structs and other data primitives have value semantics. (This includes strings, even though they are technically reference types.) But sometimes it’s useful to have reference semantics when dealing with what would otherwise be a value type. (Referencing primitives in a singleton object, for example.)

Here are two ways of doing that.

Delegates

Delegate syntax can seem a little weird–particularly when you’re working with primitives–because they’re basically typed function pointers. They look more like methods than variables.

Here’s an example:

Wrapper class using generics

My preferred way is by combining a wrapper class with C# generics. It has nicer syntax, but does require a little more setup. The results are a little clearer, in my opinion.

Here’s an example:

And there we have type-safe, reference semantics for primitives in C#. (Works for nullable types, too.)

Publisher confirms with RabbitMQ and C#

RabbitMQ lets you handle messages that didn’t send successfully, without resorting to full-on transactions. It provides this capability in the form of publisher confirms. Using publisher confirms requires just a couple of extra lines of C#.

If you’re publishing messages, you probably have a method that contains something like this:

 

But you’re out of luck if you want:

  1. A guarantee that your message was safely preserved in the event that the broker goes down (i.e. written to disk)
  2. Acknowledgement from the broker that your message was received, and written to disk

For many use cases, you want these guarantees. Fortunately, getting them is relatively straightforward:

 

 

(You’ll have to implement event handlers for acks and nacks.)

The difference between WaitForConfirms and WaitForConfirmsOrDie is not immediately obvious, but after digging through the Javadocs, it seems that WaitForConfirmsOrDie will give you an IOException if a message is nack‘d, whereas WaitForConfirms won’t.

You’ll get an IllegalStateException if you try to use either variation of WaitForConfirms without first setting the Confirms property with ConfirmSelect.

Here’s the complete code for getting an acknowledgement from the RabbitMQ broker, only after the broker has persisted the message to disk:

Java solution to Project Euler Problem 48

Problem 48:

The series, 1^1 + 2^2 + 3^3 + … + 10^10 = 10405071317.

Find the last ten digits of the series, 1^1 + 2^2 + 3^3 + … + 1000^1000.

Running time: 125 ms

Assessment: Again, very easy and fast using arbitrary-precision arithmetic. Like one of my other solutions, I didn’t limit the output to just the last ten digits in the series, but you could easily tack that on.

Java solution to Project Euler Problem 36

Problem 36:

The Fibonacci sequence is defined by the recurrence relation:

Fn = F(n-1) + F(n-2), where F1 = 1 and F2 = 1.

Hence the first 12 terms will be:

  • F1 = 1
  • F2 = 1
  • F3 = 2
  • F4 = 3
  • F5 = 5
  • F6 = 8
  • F7 = 13
  • F8 = 21
  • F9 = 34
  • F10 = 55
  • F11 = 89
  • F12 = 144

The 12th term, F12, is the first term to contain three digits.

What is the first term in the Fibonacci sequence to contain 1000 digits?

Running time:

  • Checking for a palindrome in Base 10 first: 500ms
  • Checking for a binary palindrome first: 650ms

Assessment: This problem isn’t super interesting. What I did find interesting was that changing the order of the isPalindrome() comparison resulted in a significant difference in execution times. This makes sense because there are more binary palindromes than Base 10 palindromes. For no particular reason, I expected the compiler to optimize that section so the difference wouldn’t be as stark.

I commented out the slower method so you can play with it if my explanation is unclear.

Java solution to Project Euler Problem 25

Problem 25:

The Fibonacci sequence is defined by the recurrence relation:

Fn = F(n-1) + F(n-2), where F1 = 1 and F2 = 1.

Hence the first 12 terms will be:

  • F1 = 1
  • F2 = 1
  • F3 = 2
  • F4 = 3
  • F5 = 5
  • F6 = 8
  • F7 = 13
  • F8 = 21
  • F9 = 34
  • F10 = 55
  • F11 = 89
  • F12 = 144

The 12th term, F12, is the first term to contain three digits.

What is the first term in the Fibonacci sequence to contain 1000 digits?

Running time: 36ms

Assessment: LOL.

Java solution to Project Euler Problem 22

Problem 22:

Using names.txt (right click and ‘Save Link/Target As…’), a 46K text file containing over five-thousand first names, begin by sorting it into alphabetical order. Then working out the alphabetical value for each name, multiply this value by its alphabetical position in the list to obtain a name score.

For example, when the list is sorted into alphabetical order, COLIN, which is worth 3 + 15 + 12 + 9 + 14 = 53, is the 938th name in the list. So, COLIN would obtain a score of 938 * 53 = 49714.

What is the total of all the name scores in the file?

Running time:

  • File IO: 23ms
  • 5163 names sorted: 79ms
  • Built the list of names: 154ms
  • Total runtime: 209ms

Assessment: I broke my time measurements up so I could see how fast each major piece is. I thought that file IO would be the slowest piece of the equation–I don’t have an SSD–but it isn’t. In fact, no single piece is really the bottleneck. At less than 0.3 seconds, this is, for practical purposes, instantaneous.

If I were to do this problem again–probably in C#–I would approach it the same way, but the code would look a bit cleaner, and certainly less verbose. Using higher-level data structures makes this problem pretty simple.

Java solution to Project Euler Problem 21

Problem 21:

Let d(n) be defined as the sum of proper divisors of n (numbers less than n which divide evenly into n).

If d(a) = b and d(b) = a, where a ≠ b, then a and b are an amicable pair and each of a and b are called amicable numbers.

For example, the proper divisors of 220 are 1, 2, 4, 5, 10, 11, 20, 22, 44, 55 and 110; therefore d(220) = 284. The proper divisors of 284 are 1, 2, 4, 71 and 142; so d(284) = 220.

Evaluate the sum of all the amicable numbers under 10000.

Running time: 140ms

Assessment: Like some of the other problems, I had to read this a few times before I really understood it.

Java solution to Project Euler Problem 20

Problem 20:

n! means n * (n – 1) * … * 3 * 2 * 1

For example, 10! = 10 * 9 * … * 3 * 2 * 1 = 3628800,
and the sum of the digits in the number 10! is 3 + 6 + 2 + 8 + 8 + 0 + 0 = 27.

Find the sum of the digits in the number 100!

Running time: 2ms

Assessment: Easy and fast using arbitrary-precision arithmetic.