Make your Xtext-based editor 300 times faster*

by Алекс Руис on July 10, 2012

My sincere apologies for the sensational title. Here is the fine print: With a 3-line code change, we were able to make the Protocol Buffer editor ~300 times faster when opening files in a Java project. Speed improvements may vary depending on the size of a project’s classpath.

Some background

At Google, our Eclipse projects are bigger than anything I’ve seen before: a project’s classpath can easily have 4,000 to 8,000 jar files! As a result, any performance issues, even minor ones, are greatly amplified in our environment. Frequent delays and freezes make using Eclipse a frustrating experience.

One of my team’s goals is to make Eclipse capable of handling our gigantic projects. In a effort to find performance issues (and later on fix them) we log delays in the Eclipse UI thread. In our logs we found that one of the top 5 offenders was, to my surprise, Xtext.

The problem

Before I continue, I’d like to clarify that I don’t mean to bash Xtext. IMHO, Xtext is awesome and it delivers on its promise: create a full-blown editor from a grammar definition. I consider Xtext as one of the top-3 best Eclipse projects of all time.

OK, back to our problem. According to our logs, Xtext is taking several seconds to open a document (a .proto file in our case.) We were using Xtext 2.0.3 M6 and Eclipse 3.8 M6, BTW.

The logged stack trace is the following:

java.util.zip.ZipFile.(ZipFile.java:131)
java.util.jar.JarFile.(JarFile.java:150)
java.util.jar.JarFile.(JarFile.java:114)
org.eclipse.xtext.ui.resource.XtextResourceSetProvider.computePlatformURIMap(XtextResourceSetProvider.java:68)
org.eclipse.xtext.ui.resource.XtextResourceSetProvider.get(XtextResourceSetProvider.java:49)
org.eclipse.xtext.ui.editor.model.ResourceForIEditorInputFactory.getResourceSet(ResourceForIEditorInputFactory.java:89)
org.eclipse.xtext.ui.editor.model.JavaClassPathResourceForIEditorInputFactory.getResourceSet(JavaClassPathResourceForIEditorInputFactory.java:59)
org.eclipse.xtext.ui.editor.model.ResourceForIEditorInputFactory.createResourceFor(ResourceForIEditorInputFactory.java:68)
org.eclipse.xtext.ui.editor.model.ResourceForIEditorInputFactory.createResource(ResourceForIEditorInputFactory.java:64)
org.eclipse.xtext.ui.editor.model.JavaClassPathResourceForIEditorInputFactory.createResource(JavaClassPathResourceForIEditorInputFactory.java:37)
org.eclipse.xtext.ui.editor.model.ResourceForIEditorInputFactory.createResource(ResourceForIEditorInputFactory.java:53)
org.eclipse.xtext.ui.editor.model.XtextDocumentProvider.setDocumentContent(XtextDocumentProvider.java:118)

According to XtextResourceSetProvider‘s code, when creating a Xtext document (e.g. opening a .proto file,) Xtext is reading the manifest in every single jar in the classpath. Given our projects’ huge classpaths, this scanning can take a considerable amount of time.

Since we never got a clear answer why Xtext is doing this, we had two options:

  1. Read Xtext and EMF code to understand the issue. Reading layers of undocumented code is not as bad as it sounds. Understanding the code and fixing it requires time though. Unfortunately, I don’t have it. Other projects require my immediate attention.
  2. Take an educated guess to solve the problem. Keep reading, please.

The solution

We went for the “educated guess” alternative. Given that:

  1. we found out that the proto editor works well in non-Java projects (e.g. CDT-based ones,)
  2. we suspected that this classpath scanning was necessary for custom JVM-based languages (like Xtend,) and
  3. the Protocol Buffer language does not depend on JVM types

we were pretty confident that scanning the project’s classpath was not adding any value and we could safely remove it.

The change literally required adding 3 lines of code. The beauty of Xtext is that is uses Guice under the covers. We only needed to replace XtextResourceSetProvider with another implementation of IResourceSetProvider in the UI project’s module:

  @Override public Class<? extends IResourceSetProvider> bindIResourceSetProvider() {
    return SimpleResourceSetProvider.class;
  }

And that’s it!

The following table shows the dramatic improvement in performance. Please note the ridiculous amount of time it used to take to open small files.

LOC Time to open
Before After
199 5280 ms 17 ms
19 3885 ms 6 ms
27 2033 ms 7 ms

Please note that lines of code is not the only metric that affects performance. There are other factors (e.g. scoping,) that need to be considered when measuring performance.

We didn’t lose any functionality in our editors after making this change. Syntax highlighting, content assist, scoping, hyperlinking and validation kept working normally.

Conclusion

Xtext is a great framework for creating editors for custom languages. Although it is a mature project, there is still room from improvement. In this post, I showed you how we dramatically improved the performance of our Xtext-based editor with only a few lines of code. We were able to remove the problem in record time by using a combination of data, facts and intuition.

Feedback is always welcome :)

{ 10 comments… read them below or add one }

Sven Efftinge July 10, 2012 at 8:49 am

Hi Alex,

Good job! That code is really just needed when you have to resolve platform:resource URIs, which is mainly the case for Xtext grammars. So also for Xtend or other JVM languages this is not needed.

Although it’s not that noticeable in “normal” projects, we should reduce the impact. A bug report would be good.

Reply

Alex Ruiz July 10, 2012 at 9:02 am

Thanks Sven. A bug report is on its way!

Reply

Jan Rosczak July 10, 2012 at 11:05 pm

Hi Alex,

Thanks for sharing! Can you tell me how you log delays in the Eclipse UI thread?

Reply

Alex Ruiz July 10, 2012 at 11:15 pm

Hi Jan,

I’m sorry but I can’t at this moment. That portion of code is not open source. It may not make sense to open source it since it is tied to our internal infrastructure.

Let me check if I can at least describe how we do it. If I do, I’ll be writing a follow up post.

Cheers!

Reply

Michael Vorburger July 11, 2012 at 11:15 am

Hello, I just wanted to second the interest in the “how do you log delays in the Eclipse UI thread” topic – we (TEMENOS Core Banking SW) build our own Eclipse Modeling product, and this sounds like a very useful approach to get more information re. end-user’s “it often blocks” – interested as well if you can share more about that at some point. PS: Actually, this should be built into Eclipse! ;-)

Reply

James Blackburn July 19, 2012 at 10:12 am

And I’d like to third that request. Collecting UI thread pauses automatically would be interesting

Reply

Alex Ruiz July 19, 2012 at 11:45 am

Jan, Michael, and James,

This logging tool is still under development. We noticed that we are not getting all the data that we expect, and we are still trying to figure out what we’re missing. I think it would be better to talk about it once we get something working reliably. Stay tuned.

Cheers!

Reply

Aaron Digulla September 20, 2012 at 12:11 am

Here is a better implementation of XtextResourceSetProvider which caches information for JAR files: https://gist.github.com/3754363

Reply

Alex Ruiz September 20, 2012 at 8:55 am

Great! I think it would be a good idea to submit your implementation to the Xtext project. WDYT?

Reply

Ozan Aksoy February 15, 2013 at 12:15 am

Hi, This is a very interesting post. I would like to give it a try.

What I understand from your post: Let’s say I have a org.xtext.example.sample with Sample.xtext project, I would need to add:

@Override public Class bindIResourceSetProvider() {
return SimpleResourceSetProvider.class;
}

to: org.xtext.example.sample.ui/src/org/xtext/example/sample/SampleUIModule

and that’s it! I hope there won’t be any issues.

Reply

Leave a Comment

Previous post:

Next post: