I managed to get a number of problems solved this weekend, mostly dealing with the factory model application. My number one priority was to create a search facility for the factory web application. I am still using apache Solr as a webservice for the search.
Last Wednesday I tried to create a Vaadin addon that would use a Solr backend to create a search widget. This was my first real addon, and perhaps I should have tried something simpler as a first attempt. I did get a lot of experience in debugging the addons. The basic addon project template wants to use Jetty as the web server for the server side code, which sort of disrupts the normal flow. In order to run the addon in the GWT shell, you first have to deploy it. I'm not sure what that buys me, as normally I would use the GWT shell's built-in Tomcat server. Perhaps at some point I will switch the configuration back to using the built-in Tomcat, but for now I just left it as it is, trying not to change too much in the project it setup for me.
The Vaadin addon was based on GWT's SearchBox and SearchOracle, which I have used successfully in the past for this kind of search. The SearchOracle has callbacks that are invoked whenever a key is pressed in the search box to provide a list of matches that is refined as the user types. It uses the Solr webapp directly via GWT's requestBuilder utility class. The only issue was that the request always returned a status of 0 rather than a status of 200 or some other normal HTTP code. Googling that situation only lead to sites that indicate that status 0 is a reasonable code only if the protocol is something other than http or https. I could see the query in the Solr log, so I was certain that 1) the request was http 2) the request was making it to the server and 3) the server was returning 200. My working theory is that Vaadin somehow messes with the workings of the requestBuilder, which relys on ajax calls to fulfill the request. I posted a question on the Vaadin forum but have yet to receive a response. A new theory that I'm just now realizing is that SOP (same origin policy) is interfering, since the Solr service is running on a different port than the Vaadin test server. I will follow up on that after I post.
Taking a different approach on Saturday, I tried creating a search widget using standard Vaadin components, and having the search running on the server side. The disadvantage of that is that there is an extra http call introduced per keystroke. Since Vaadin doesn't have anything similar to the SearchBox/SearchOracle, I would have to create a composite widget that would combine a text field and a list box. The only hard part is that I need the list box to pop up as soon as the user started typing, and disappear when the user selected something. I toggled the visible flag to make it appear/disappear, but it would displace the rest of the form, and I needed it to pop up over the rest of the form. Vaadin doesn't seem to provide a simple pop up, only one that's formatted as a window, with a title bar and other unwanted formatting. To do what I wanted and work within the Vaadin framework, I need to use CSS.
It was time to bite the bullet and introduce 'themes' to my application. Themes is the Vaadin way of adding CSS and images to a webapp. If you don't have a theme, a standard one is used without any intervention. Adding a theme requires that you pull the standard theme out of the Vaadin jar, and put it into your project, so that your new theme can derive from the standard theme. Then you set your application to use the theme with the setTheme method.
Themes go in a specific folder 'VAADIN/themes' and contain at least one stylesheet 'styles.css' (don't forget to make it plural, I had some trouble when Vaadin couldn't find it because it was named style.css). To add styles to a component you use the addStyleName method which will add your style to the list of styles. I created styles for the composite box, the text field, the list and a button. I also tried to include a label for the text field. That was one of my first issues, Vaadin always puts labels above the text field, and I wanted it next to the field. Using the Chrome developers tools I was able to inspect the DOM and found that Vaadin wraps almost everything in <div> tags, which stack by default. After being unable to figure out how to get the label in a <span> tag, I discovered that CSS now can override the <div> tag's stacking behavior with the 'inline' display attribute. When they still stacked after setting the tags to be inline, I realized that the enclosing space wasn't large enough, so I had to set that. Then on to trying to get the pop up list to not effect the rest of the form. That is done by setting the positioning to be absolute, and positioning it relative to the enclosing box. After a few hours of trial and error, I finally managed to get something halfway presentable.
Now I can go on a tangent about what a mess the state of CSS is in. If there's a reason that good web programmers are in such demand, it's because of the steep learning curve to overcome in using HTML and CSS to create decent web pages. The WYSIWIG designers produce horrendous code that is all but in-decipherable and almost completely unmaintainable, so unless you're working on a one-off project, steer clear. The reason is it's incredibly difficult to translate a reasonable graphical layout to consistent CSS/HTML, while overcoming all the browser quirks. It still requires the time of a skilled web programmer to create a rational structure that can be understood and maintained. So I continue to increase my skills in CSS because there's no way out of it. Even with GWT or Vaadin, there's no good way to program yourself out of dealing with CSS.
Saturday I finished getting the search box to display a basic query, and left it at that. There still was a process issue, my stack now has all these components, web services, database, Solr search service. I need some way of coordinating all those components. Additionally, the Solr search service wasn't configured correctly, so it was only returning a subset of items.
This morning, I focused on those two issues. First of all the Solr search service. A wide open search was only returning a small subset of what it was supposed to. Looking at the subset, I could tell that they were only indexing stockrooms and shops, which are both derived from 'internal organization'. That happened to be the last entity described in the DataImportHandler database config xml. The problem was that it was using the id field as a unique key across all types. At least with the MySQL implementation, and the standard DDL generated by JPA, the id was not unique across types, so the part table had id's 1, 2, 3... and the internalorganization table had id's 1, 2, 3... as well. I had to create a key that was unique across the whole database. The solution was to create a special field in the select: "concat(id,'-type') idtype" substituting the actual type for type in '-type'. That created a field called idtype which I could use for all entities.
The second issue I solved was one of process. I needed a process that would encapsulate creating the database, loading data, deploying the webservice API and the seach service. I have been evolving my thinking on Maven as a build tool. While it still is the only tool I would consider for putting together 'things', it is sorely deficient on encapsulating 'actions'. Each POM is focused on creating a 'thing'. When you need it to perform some action that is above an beyond what is needed for creating a thing, you are cutting against the grain. So I'm back to my old friend Ant to perform the actions that I need done once all my artifacts are created. I had an Ant script that I used to startup the search service, that seemed like a good place to start. There are a set of Maven Ant tasks that allow you to use artifacts in a Maven repository in a fileset and a classpath. I used both, as I used the classpath generated by my ERPLoader module to run the database loader step, and I used the fileset of the ERPWebservice module to copy the war file to the Solr instance of Jetty. Now my process was performed with a single ant build.
As my thinking has evolved with Maven, I have been looking at other build tools, most notably Buildr. There has been a lot of enthusiasm with Buildr, especially it's simplicity compared with Ant and Maven. But I have a real problem with having to learn yet another expression language (Ruby) just to do what I'm already comfortable with. I suppose it makes sense for a Ruby programmer to use Buildr, especially for simple cases. However, in addition to the problem I have with it being in Ruby, there are two other issues that that make me favor Maven in most cases and Ant in the remaining cases. Firstly, Ruby may be simpler for people to read, but it's not simpler for computers. Secondly, (and Ant suffers from this too) it is procedural, rather than descriptive, so it's much harder for third party products to infer anything from them. These two things mean that all the Maven adaptive applications such as Netbeans and Hudson, are less able to adapt to Buildr builds and Ant builds. Things such as dependency maintenance are a breeze using Netbeans, using point-and-click tools to find and configure my dependencies. In this respect, Ant and Buildr are too flexible for their own good, while Maven with it's admittedly rigid structure shines. I think the trick to using Maven and not falling into the hole of mile long pom files, is to separate out actions that are intended to build artifacts with actions that are intended to use artifacts. Use Maven to build artifacts, but use Ant or some other scripting language when you're in use mode, since the act of building tends to be pretty straight forward, but the act of using is much more open-ended, and requires a lot more flexibility.
Monday, May 30, 2011
Sunday, May 8, 2011
3D Graphics in a browser and a reconciliation between JPA and JAXB
After the week's success with Vaadin, I did some research into running 3D Graphics in a browser, and got around to adding XML binding to my factory model this weekend. That in addition to some gardening Saturday afternoon, my final choir concert Saturday evening and going to see the Seattle Symphony this afternoon.
A few months ago on the Google code blog, they demonstrated a port of Quake II to GWT, and had it running in a browser. That's right, a 3D real time game running in a browser without any plugins (not even Flash). HTML5 which is semi-supported by Chrome, Safari, and Firefox includes WebGL which is based on OpenGL. WebGL is a canvas based drawing/rendering system that exposes a JavaScript API for creating 2D and 3D graphics. I had some interest in this, as I still have parts of a 3D networked multiplayer that could make use of this.
My scope for my research was pretty simple: could I create a very simple Maven/GWT project that would create a simple 3D webapp. I started with a basic Maven/GWT webapp project from an archetype I was able to follow the instruction, although I added the step of temporarily removing my local repository mirror since I was dealing with a foreign repository, and I didn't want the mirror to interfere. From there I went to the GwtGL site, which is basic GWT bindings for the WebGL interface. It had two section, how to add what you need to your webapp project to use GwtGL and a sample web app that just displays a white triangle in a black square.
Adding GwtGL was pretty straight forward. Add a div tag to be your canvas, inherit the library in the GWT project descriptor and include the library (and source) in your classpath. Before I could do anything, I needed to download the GwtGL source and binaries and install it in my local repository, as it's not on Maven Central yet. I'm using Sonatype Nexus as a repository manager, and it has a nice artifact upload interface. The standard setup for Nexus includes a 3rd party repository for uploading 3rd party artifacts that have yet to make it to Maven Central. In the zip file download from GwtGL is the binary, source and javadoc jar plus the POM file, and I uploaded all of them to Nexus. For some reason the groupId is com.cooglecode, I'm not sure what that's about. Then I added to the main.html the div tag:
<div id="gwtGL"'></div>
The inheritance tag goes in the GWT project descriptor:
<inherits name='com.googlecode.gwtgl.binding'/>
And finally the binary and source jars need to be added to the dependencies:
<dependency>
<groupId>com.cooglecode.gwtgl</groupId>
<artifactId>gwtgl</artifactId>
<version>0.3</version>
</dependency>
<dependency>
<groupId>com.cooglecode.gwtgl</groupId>
<artifactId>gwtgl</artifactId>
<classifier>sources</classifier>
<version>0.3</version>
</dependency>
The sample code had one issue, it would appear that a class name has changed, but the sample code wasn't updated. After a bit of poking around, I discovered the updated class name for WebGLFloatArray was Float32Array. A quick compile, and run and I had a browser with a black square with a white triangle. Maybe next weekend I'll look at what it will take to convert the 3D game client to run in a browser.
Today I spent some time making an export and import API for the static objects in my factory model. I wanted to be able to import and export to XML all of the parts and build steps, shops and stockrooms that comprise what I call the 'factory matrix'. This is everything that has a static description which isn't supposed to change, and so would probably loaded once into a database and rarely changed. In order to be able to do this with JAXB, I needed to break any cyclic links or bi-directional references. I decided to use the parent/child reference technique where I set the parent reference to transient with the XMLTransient annotation, so it's not set in the initial traversal, but set later on with a callback called afterUnmarshal. The only issue I had was that the XMLTransient annotation needs to go on the bean getter, not on the field itself. I had it on the field, and was still getting cyclic reference exceptions when marshaling the factory matrix. Putting it on the getter, and I was back to 'build success' which always makes my heart happy. In addition to the import and export function, I added a lot of unit tests, hopefully my code coverage reports won't look so dismal tomorrow after new reports are generated.
A few months ago on the Google code blog, they demonstrated a port of Quake II to GWT, and had it running in a browser. That's right, a 3D real time game running in a browser without any plugins (not even Flash). HTML5 which is semi-supported by Chrome, Safari, and Firefox includes WebGL which is based on OpenGL. WebGL is a canvas based drawing/rendering system that exposes a JavaScript API for creating 2D and 3D graphics. I had some interest in this, as I still have parts of a 3D networked multiplayer that could make use of this.
My scope for my research was pretty simple: could I create a very simple Maven/GWT project that would create a simple 3D webapp. I started with a basic Maven/GWT webapp project from an archetype I was able to follow the instruction, although I added the step of temporarily removing my local repository mirror since I was dealing with a foreign repository, and I didn't want the mirror to interfere. From there I went to the GwtGL site, which is basic GWT bindings for the WebGL interface. It had two section, how to add what you need to your webapp project to use GwtGL and a sample web app that just displays a white triangle in a black square.
Adding GwtGL was pretty straight forward. Add a div tag to be your canvas, inherit the library in the GWT project descriptor and include the library (and source) in your classpath. Before I could do anything, I needed to download the GwtGL source and binaries and install it in my local repository, as it's not on Maven Central yet. I'm using Sonatype Nexus as a repository manager, and it has a nice artifact upload interface. The standard setup for Nexus includes a 3rd party repository for uploading 3rd party artifacts that have yet to make it to Maven Central. In the zip file download from GwtGL is the binary, source and javadoc jar plus the POM file, and I uploaded all of them to Nexus. For some reason the groupId is com.cooglecode, I'm not sure what that's about. Then I added to the main.html the div tag:
<div id="gwtGL"'></div>
The inheritance tag goes in the GWT project descriptor:
<inherits name='com.googlecode.gwtgl.binding'/>
And finally the binary and source jars need to be added to the dependencies:
<dependency>
<groupId>com.cooglecode.gwtgl</groupId>
<artifactId>gwtgl</artifactId>
<version>0.3</version>
</dependency>
<dependency>
<groupId>com.cooglecode.gwtgl</groupId>
<artifactId>gwtgl</artifactId>
<classifier>sources</classifier>
<version>0.3</version>
</dependency>
The sample code had one issue, it would appear that a class name has changed, but the sample code wasn't updated. After a bit of poking around, I discovered the updated class name for WebGLFloatArray was Float32Array. A quick compile, and run and I had a browser with a black square with a white triangle. Maybe next weekend I'll look at what it will take to convert the 3D game client to run in a browser.
Today I spent some time making an export and import API for the static objects in my factory model. I wanted to be able to import and export to XML all of the parts and build steps, shops and stockrooms that comprise what I call the 'factory matrix'. This is everything that has a static description which isn't supposed to change, and so would probably loaded once into a database and rarely changed. In order to be able to do this with JAXB, I needed to break any cyclic links or bi-directional references. I decided to use the parent/child reference technique where I set the parent reference to transient with the XMLTransient annotation, so it's not set in the initial traversal, but set later on with a callback called afterUnmarshal. The only issue I had was that the XMLTransient annotation needs to go on the bean getter, not on the field itself. I had it on the field, and was still getting cyclic reference exceptions when marshaling the factory matrix. Putting it on the getter, and I was back to 'build success' which always makes my heart happy. In addition to the import and export function, I added a lot of unit tests, hopefully my code coverage reports won't look so dismal tomorrow after new reports are generated.
Saturday, May 7, 2011
An Aha! coding moment
The Google Web Toolkit (GWT) has held the promise of web-programming nirvana since it's introduction a few years back. Yet in practice, it has fallen short of it's heavenly goals for several reasons. First of all, it adds minutes to a code-compile-test cycle that coders have gotten used to taking seconds. Debugging in the GWT shell requires the patience of a saint. And lastly, it forces an awkward split between the code that runs on the browser and the code that runs on the server.
Admittedly, most of these problems and more exist in traditional web-programming environments, which is why I've always considered the browser-based apps as a huge step backwards in user interfaces, both in terms of coding them and using them. It also means that anyone with the wherewith-all to do web coding are the stars of the coding world. Everything is moving to the web, from games to business app, and if you're not on the band-wagon, your going to be the COBOL coders of the modern era.
GWT was supposed to solve all that by allowing all web coding in Java, there-by allowing the great abundance of Java tools to be used, and anyone with Java skills to become a web-programming star. It was genuinely easy to create the ubiquitous 'hello world' web app and have it up on your browser in seconds. The problem is that it didn't scale. The minute you tried to do anything real with it, the compile time started taking forever. You couldn't split up your code and have some of it pre-compiled, everything that ran on the browser side had to be compiled in one step. Add to that the lack of libraries and features that Java programmers have gotten used to over the years. You couldn't use one of the many XML libraries available, you were stuck with a very basic XML parser. There wasn't a tradition 'Class' object so no introspection, or dynamic object creating, which is the main-stay of anyone that is trying to separate out application code from architectural code. One way of over-coming much of this was to move code over to the server side which had access to the complete richness of the Java language and libraries. In all MVC architectures that I've seen thus far, this was done by splitting the model side and providing a synchronization layer between the client and the server.
Then, last week I was reading on the Archiva developers mail list about a planned refactoring, which would include the UI. It was suggested to use GWT for the UI, but Vaadin was also mentioned as a possibility. I had not heard of Vaadin before, so I did a quick google search for it and looked through some sample code that they had. It looked a lot like a GWT app, a module that was loaded at the start, creating a UI with java classes, a simple model binding for the UI. I was amazed when I downloaded it and compile it, it compiled in seconds, with only the standard Java compiler, and yet it ran like a full browser app. Where was the GWT compiler? How was all this UI code running in the server? There was no messy configuration, no limitations on the Java code. It was like compiling a simple Swing program, but it ran in the browser, just like a GWT program.
I had to find out what the magic was behind all this. After digging around a bit, I discovered that they had made a fundamental shift in the MVC architecture. Instead of splitting the model and keeping it synchronized, they split the client. Aha! all of a sudden, the possibilities became apparent. The Vaadin library had come from a different place, by creating an abstract terminal model for the browser prior to the advent of GWT. When GWT came along they adopted it into their model. Now we could have a GWT app, with most of the programming done in the server. The only time you needed to deal with GWT is when creating custom widgets.
I wanted to put Vaadin to use, and I had been working on my factory model application during my spare time for several week prior (as I mentioned in my last post on JAXB/JPA). I decided to created a GUI front-end for that model with Vaadin. In almost no time, I adapted their form example to my part object, so now I had a simple CRUD (Create-Read-Update-Delete) form for parts. Then, I adapted that form for stockroom, with equal ease. I factored out a basic CRUD form, and then created forms for all of the static objects in the factory model. I added drop-down lists for all one-to-n relationships that had a reasonably small n-count, and the dual-list chooser for n-to-n relationships. Basically, I had created a substantial fraction of an entire business application in just a few spare hours over a few days. The great part was that it worked without coercion directly with all my JPA objects. Here was a library that 'just works' with JPA. Most of the nuances of servlet programming were invisible to me (everything ran inside a session). And I had none of the issues of dealing with the browser.
The Aha moment wasn't actually mine, it was a 'domino' Aha moment. Obviously the people at Vaadin (which is a Finnish word for a semi-domesticated adult female reindeer) had the initial moment when they had the fore-sight to tame the web side by creating the abstract terminal model. They then embraced GWT as the language of choice for creating the web side of the terminal model. For me, the Aha moment was the sudden realization of all the possibilities that this confluence enabled.
I will continue to work with this library to see how well it does for other typical web application stumbling blocks such as grids for large tables, and editable grids. It has been a pleasure to work with so-far, hopefully it scales well, and is easily adaptable. If it continues to make coding fun, it will certainly become my primary means of coding for the web. I think that basic GWT still has a place, especially for browser applications that don't conform to the terminal model.
Admittedly, most of these problems and more exist in traditional web-programming environments, which is why I've always considered the browser-based apps as a huge step backwards in user interfaces, both in terms of coding them and using them. It also means that anyone with the wherewith-all to do web coding are the stars of the coding world. Everything is moving to the web, from games to business app, and if you're not on the band-wagon, your going to be the COBOL coders of the modern era.
GWT was supposed to solve all that by allowing all web coding in Java, there-by allowing the great abundance of Java tools to be used, and anyone with Java skills to become a web-programming star. It was genuinely easy to create the ubiquitous 'hello world' web app and have it up on your browser in seconds. The problem is that it didn't scale. The minute you tried to do anything real with it, the compile time started taking forever. You couldn't split up your code and have some of it pre-compiled, everything that ran on the browser side had to be compiled in one step. Add to that the lack of libraries and features that Java programmers have gotten used to over the years. You couldn't use one of the many XML libraries available, you were stuck with a very basic XML parser. There wasn't a tradition 'Class' object so no introspection, or dynamic object creating, which is the main-stay of anyone that is trying to separate out application code from architectural code. One way of over-coming much of this was to move code over to the server side which had access to the complete richness of the Java language and libraries. In all MVC architectures that I've seen thus far, this was done by splitting the model side and providing a synchronization layer between the client and the server.
Then, last week I was reading on the Archiva developers mail list about a planned refactoring, which would include the UI. It was suggested to use GWT for the UI, but Vaadin was also mentioned as a possibility. I had not heard of Vaadin before, so I did a quick google search for it and looked through some sample code that they had. It looked a lot like a GWT app, a module that was loaded at the start, creating a UI with java classes, a simple model binding for the UI. I was amazed when I downloaded it and compile it, it compiled in seconds, with only the standard Java compiler, and yet it ran like a full browser app. Where was the GWT compiler? How was all this UI code running in the server? There was no messy configuration, no limitations on the Java code. It was like compiling a simple Swing program, but it ran in the browser, just like a GWT program.
I had to find out what the magic was behind all this. After digging around a bit, I discovered that they had made a fundamental shift in the MVC architecture. Instead of splitting the model and keeping it synchronized, they split the client. Aha! all of a sudden, the possibilities became apparent. The Vaadin library had come from a different place, by creating an abstract terminal model for the browser prior to the advent of GWT. When GWT came along they adopted it into their model. Now we could have a GWT app, with most of the programming done in the server. The only time you needed to deal with GWT is when creating custom widgets.
I wanted to put Vaadin to use, and I had been working on my factory model application during my spare time for several week prior (as I mentioned in my last post on JAXB/JPA). I decided to created a GUI front-end for that model with Vaadin. In almost no time, I adapted their form example to my part object, so now I had a simple CRUD (Create-Read-Update-Delete) form for parts. Then, I adapted that form for stockroom, with equal ease. I factored out a basic CRUD form, and then created forms for all of the static objects in the factory model. I added drop-down lists for all one-to-n relationships that had a reasonably small n-count, and the dual-list chooser for n-to-n relationships. Basically, I had created a substantial fraction of an entire business application in just a few spare hours over a few days. The great part was that it worked without coercion directly with all my JPA objects. Here was a library that 'just works' with JPA. Most of the nuances of servlet programming were invisible to me (everything ran inside a session). And I had none of the issues of dealing with the browser.
The Aha moment wasn't actually mine, it was a 'domino' Aha moment. Obviously the people at Vaadin (which is a Finnish word for a semi-domesticated adult female reindeer) had the initial moment when they had the fore-sight to tame the web side by creating the abstract terminal model. They then embraced GWT as the language of choice for creating the web side of the terminal model. For me, the Aha moment was the sudden realization of all the possibilities that this confluence enabled.
I will continue to work with this library to see how well it does for other typical web application stumbling blocks such as grids for large tables, and editable grids. It has been a pleasure to work with so-far, hopefully it scales well, and is easily adaptable. If it continues to make coding fun, it will certainly become my primary means of coding for the web. I think that basic GWT still has a place, especially for browser applications that don't conform to the terminal model.
Subscribe to:
Comments (Atom)