Category Archives: Software

dday.ical is now ical.net and available under the MIT license with many performance enhancements

ical.net, Programming, Software

A few months ago, I needed to do some calendar programming for work, and I came across the dday.ical library, like many developers before me. And like many developers, I discovered that dday.ical doesn’t have the best performance, particularly under heavy server loads.

I dug in, and started making changes to the source code, and that’s when I discovered that the licensing was ambiguous, and that it had been abandoned. I was concerned that I might be exposing my company to risk due to unclear copyright, and a non-standard license.

With some effort, I was able to track down Doug Day (dday), and he gave me permission to fork, rename (ical.net), and relicense his library (MIT), which I have done. So I’m happy to report…

dday.ical is now ical.net

mdavid, who saw to it that the library wasn’t lost to the dustbin of Internet history, has graciously redirected dday users to ical.net. Khalid Abuhakmeh, who published the dday nuget package that you might be using (you should switch ASAP) has also agreed to archive and redirect users to ical.net.

So… why should you use the new package?

Unambiguous licensing

Doug has revoked his copyright, and given unrestricted permission to give dday.ical new life as ical.net. That means ical.net is unencumbered by legal ambiguities.

Many performance enhancements

My changes to ical.net have been mostly performance-focused. I was lucky in that dday.ical has always included a robust test suite with about 170 unit tests that exercise all the features of the library. Some were broken, or referenced non-existent ics files, so I nuked those right away, and concentrated on the set of tests that were working as a baseline for making safe changes.

The numbers:

  • Old dday.ical test suite: ~17 seconds
  • Latest ical.net nuget package: 3.5 seconds

There’s no games here. ical.net really is that much faster.

Profiling showed a few hotspots which I attacked first, but those only bought me maybe 3-4 seconds improvement. There was no single thing that resulted in huge performance gains. Rather it was many, many small changes that contributed, quite often by improve garbage collection pauses, many of which were 5ms+, which is an eternity in computing time.

Here are a few themes that stand out in my memory:

  • Route all time zone conversions through NodaTime, which actually exposed some bugs in what the unit tests were asserting
  • Converting .NET 1.1 collections (Hashtable, ArrayList) to modern, generic equivalents
  • Converting List<T> to HashSet<T> for many collections, including creating stable, minimal GetHashCode() methods, though more attention is still needed in this area. A nice side effect of this was that lot of lookups and collection operations then became set operations (ExceptWith(), UnionWith(), etc.)
  • Converting several O(n^2) methods to O(n) or better by restructuring methods based on information that was available in context
  • Converted a lot of loops to LINQ. (Yes, really!)
  • Specifying initial collection sizes when using array-backed collections like List<T> and Dictionary<TKey, TValue>
  • Moved variables closer to their usage, which sometimes meant that certain expensive calls don’t occur at all, because the method exits before reaches it. This also had the effect of pushing some variables into gen 0 garbage collection. (Anecdotally, I have noticed GC pauses are fewer and further between, though I don’t have any hard data that it’s actually significant.)
  • Moving expensive calls outside of tight loops. Unfortunately the library makes extensive use of the service-provider antipattern. A common thing was to have an expensive call (get me a deserializer for Foo!) inside a tight loop that’s only ever deserializing Foos. So you can make the call once and just reuse the deserializer.
  • Implemented a lazy caching layer as suggested in one of the TODOs in the comments.

Along the way, I converted a lot of code to modern, idiomatic C#, which actually helped performance as much as any of the discrete things I did above. As I work towards a .NET Core port, I have the runtime down to about 2.8 seconds just through clarifying and restructuring existing code, and idiomatic simplifications.

What’s next?

  • A .NET Core port is nearly complete.
  • The ical.net has virtually no documentation. I hope to improve the readme with some simple examples this morning/afternoon.
  • I have been bug collecting on Stack Overflow, and have a few maybe-bugs to investigate and/or write test cases for.
  • Maybe some API changes for v3, still TBD. I’ll discuss these in a future blog post.

Creating an array of generics in Java

Software, , ,

I was messing around with creating a generic Bag collection in Java that’d be backed by an array. It turns out that you can’t do this for a number of interesting reasons…

In Java (and C#), arrays are covariant. This means that if Apple is a subtype of Fruit, then Apple[] will also be a subtype of Fruit[]. Pretty straightforward. That means this will compile:

Apple[] appleArray = new Apple[10];
appleArray[0] = new Apple();
Fruit[] fruitArray = appleArray;    //Spot the problem?

If you’re like me, you didn’t think too hard about this, and assumed you could do the same with parameterized types, i.e. generics. Thanksfully you can’t, because that code is unsafe. It will throw an ArrayStoreException at runtime which we’d have to handle.

Wouldn’t it be great if we could guarantee type safety at compile time?

Generics are safer

Unlike arrays, generics are invariant, which means that Apple being a subtype of doesn’t matter: a List<Apple> is different than a List<Fruit>. The generic version of the code above is illegal:

Vector<Apple> apples = new Vector();
Vector<Fruit> fruits = apples;       //Compile-time error

You can’t cast it, either:

Vector<Apple> apples = new Vector();
Vector<Fruit> fruits = (Vector<Fruit>)apples; //Still a compile-time error!

By making generics invariant, we guarantee safe behavior at compile time, which is a much cheaper place to catch errors. (This is one of the big reasons developers get excited about generics.)

So why are arrays and generics mutually exclusive?

In Java, generics have their types erased at compile time. This is called type erasure. Type erasure means a couple of things happen at compile time:

  • Generic types are boiled down to their raw types: you cannot have a Derp and a Derp<T> in the same package.
  • A method that has a parameterized type overload won’t be compile: a class with methods popFirst(Derp<T> derp) and popFirst(Derp derp) won’t compile.
  • Runtime casts are inserted invisibly by the compiler to ensure runtime type safety. (This means there’s no performance benefit to generics in Java!)

Java’s implementation of generic types is clumsy, and was done to maintain backward-compatibility in the bytecode between Java 5 and Java 4.

Other high-level languages (like C#) implement generics very differently, which means none of the three caveats above apply. Generics in full-stack implementations do net performance gains along with those type-safety guarantees.

To recap, in Java:

  • Arrays require type information at compile time
  • Generics have their types erased at compile time

Therefore you cannot create arrays of parameterized types in Java.

Further reading

How to fix broken iCloud photostream sync on Windows

Software, , ,

Symptom

  • Your iPhone is set to back up your photos to iCloud
  • iCloud on your Windows machine is configured to download your photos
  • iCloud isn’t downloading your photo stream.

Fix

  1. Open the Task Manager by hitting Ctrl+Shift+Esc
  2. Click the Processes tab
  3. Click Name to sort the processes by name
  4. Find the Apple Photostreams Uploader and Apple Photostreams Downloader processes. End both of them.
    • In Windows 7, these will be called ApplePhotostreamsUploader.exe and ApplePhotostreamsDownloader.exe
  5. Hold down your Windows key, and hit R to open a Run prompt
  6. Type %appdata% and hit Enter
  7. Open Apple Computer > MediaStream
  8. Delete everything in the directory
  9. Log out of your Windows account, and log back in (or just reboot, if you find that easier)
  10. Once you’ve logged back into your Windows account, open the iCloud control panel again
  11. If the Photos checkbox is empty, check it
  12. Click Options, and make sure the photo options are configured how you want them
  13. Click Apply

In a few moments, your photos should start downloading.

Notes

iCloud isn’t very smart about a great many things. Here are a few:

  • If you changed the location of your downloaded photos, it will redownload what it can, creating duplicates.
  • In the iCloud 2.x days, your downloads and uploads were usually split into a Downloads and Uploads directory, and you could change the directories if you wanted. That’s not true anymore. Instead, iCloud 3.x creates a “My Photo Stream” directory, and sticks your downloads in there. Anything you’ve shared with other people, or that other people have shared with you goes into “Shared”. If you want to push a photo from your computer to iCloud, put it into Uploads


If you found this post useful, please consider donating $2