Rethinking JPA

For several years now I have been using JPA at both work and home.  I've been evolving my thinking about modeling with Java objects that are backed by JPA, and how to make that work most naturally.  Much of the trouble stems from the fact that object modeling and database modeling take very different strategies, even if they end up with the same basic model.  JPA is supposed to hide the details of the database model from you, but it always seems to let some of that leak out.  What I will layout here is a new approach that I am trying out in my factory model system.
One of my home projects is a factory modeling system.  It was created because I wanted to explore how to best model a manufacturing plant.  A lot of the software that I create at work also deals with modeling a manufacturing plant, but I wanted to create my own that was free of a lot of the historical artifacts that infect the work system.  My home project can be changed at a whim as my thinking evolves, something that could not be done at work for obvious reasons.  In some of my other projects I had stumbled upon a new way of thinking about JPA, and I wanted to try it out on the factory modeling system.  It turned out to be a major "aha" moment in my thinking about the factory model.
In a normal object model, I will most likely tie all the objects together in some sort of "universe".  The universe contains the pieces and describes the global aspects of the model.  If you want to have a model backed by a database, often it is because of the scope of the model, so for instance, the factory model might have millions of part descriptions, either purchased or built, so having a universe in the traditional sense would not be feasible.  So my original factory model had no universe, only discreet pieces that had no context, and were assumed to be global.
But the system needed a universe, if only to provide context for each of the pieces.  My recent experience showed that things were much simpler if a universe was provided, so I looked at the factory model to try to figure out what I could use as a universe.  Of course, what I came up with was a factory object that contained all of the pieces that I had so far in my model (not to be confused with the factory pattern).
The first problem with having a factory object, is normally I would want it to contain lists of all the sub-objects.  That could be massive, and I didn't want to impose a limit on this system, even if it never is used in a massive scenario.  Creating a limited system buys me no experience in creating a scale-able solution.  I did it anyway, just to try it out.  So my factory contained a number of 1-n relationships:

@ManyToOne(cascade=CascadeType.ALL)
List<Part> parts;

and then adding a part became a part of the factory class:

public Part addPart(String partName) {
     Part ret = new Part(this, partName) {
     parts.add(ret);
     return ret;
}

This moved it away from being a static method of the Part class.  Now the only static methods were for adding and finding the factory class.  The nice thing about having a universe class, is now you can model at a level above the universe if need be.  Making sure that the Factory wasn't a singleton, now I am free to model a company that has several manufacturing plants just by adding the next layer of universe (something that I had wanted to do originally but left it off for simplicity sake).
My problem of scale-ability remained, this system couldn't be used in a real setting due to it's limitation of maintaining a list of parts in the factory object.  JPA has many advanced features of lazy-loading that might prevent that in many or most cases, but worst-case scenario was both possible and unacceptable. In addition, that means giving up control to the assumption that JPA will do what's best, which is good enough for simple cases, but not for production-ready systems.  The problem dissolved away because I had the access to all of these pieces hidden behind the facade of the factory object.  It was a simple matter of removing the List objects of the factory anywhere that they might be large (in my case I simply removed all of them), and then fix up any methods in the factory class to use JPA to substitute for the list.  So my add function became:


public Part addPart(String partName) {
     Part ret = new Part(this, partName) {
     EntityManagerContainer().getEntityManager().persist(ret);
     return ret;
}

I think the key is that you create the model fully implemented as Java objects, test it out on a small scale, and then identify the lists that might be too large to be easily instantiated, remove the lists and fix the methods as shown above.
After a lot of editing, this has worked beautifully, and simplified my tests and my usages of the system.  I have already refactored my Vaadin GUI to use the new system, and it fit right into place.  Next step, maybe I will try to put all of my AI beings from another project to work in a new factory, and see if they create anything.  All code can be found here: https://www.kamradtfamily.net/svn/projects/trunk/erpsystem and here: https://www.kamradtfamily.net/svn/projects/trunk/erpgui

A new paradigm to conquer

At long last I've decided to investigate Git as a version control system.  I've converted one of my projects to Git using github.com, and managed to get it to build and release using Jenkins.  Much help was found here: http://www.nomachetejuggling.com/2011/07/31/ubuntu-tomcat-jenkins-git-ssh-togethe/ I found myself in the middle of a big debate however, about the nature of distributed version control systems and their advantages and disadvantages.  Much of the debate can be found here: http://jira.codehaus.org/browse/SCM-444 It would seem I've stepped into a mine-field of issues centered around Git's distributed flexibility, and Maven's rigid stability.

The issues call into question a lot of closely held beliefs I have about reproducibility and stability, and the purpose of continuous integration.  The crux of the issue is that Git enforces no central 'authoritative' server, but allows an entire repository to be cloned locally, and then only reconciled when desired.

One of the first advantages I can see in a distributed system like Git is an issue that I'm currently facing at work: I have a lot of changes that need committing, but I still have some work to do before it's ready to share.  Git solves this by separating the committing and sharing process, so that I can commit my work more frequently, and have finer granularity if I choose to back some of that work out.  In my current situation at work, as I progress through the changes necessary to finally be able to share my work, If I should need to back out just the last hours work, I have no recourse. I could make patches as checkpoints (and I think I'll do that first thing tomorrow), but that would be actions that I am not accustomed to, and could be error prone.  Git would make this a trivial problem, by allowing me to commit to a local repository, and finally share when it's totally ready.

One of the disadvantages of a distributed system like Git is that it enforces no policy on sharing, and so leaves that up to the project owner/committers.  I suppose this is somewhat akin to the sticking point many have about SVN, that there is no enforced notion of tag/branch, so it becomes up to committers and convention.  And there are many tools that assume one convention or another concerning SVN branches and tags.  So is true about Git and sharing, there are tools that make assumptions about the conventions adopted by project owners. Specifically, when Jenkins wants to release a version, it must share some changes to the project, so that the version numbers are maintained consistently.  This assumes there is some centralized authoritative repository to share to.

So to be able to maintain versions consistently, one needs to adopt a sharing policy, specifically a centralized authoritative repository where the versions can be stored at release time.  One person described the problem as an impedance mismatch, because Git uses a hash value as an authoritative version, and Maven uses a natural sequential number.  I suppose that, most logically, releasing a version in Maven would cause a branch to be created in Git, but that would still mean that after every release, you would have to clone another branch on your machine to continue working on the latest version.

I see the advantages to a distributed system like Git, but in order to not allow sharing to become a free-for-all, some control must be established.  I see that control in the Maven release process, and control means limiting the full capabilities temporarily and establishing a sharing policy.  Pulling together all the changes that need to go into a release, making sure everyone is in sync, releasing, and then allowing people to become asynchronous again.

The point in having a continuous integration build is to pull together what everyone has shared so-far, and making sure they work together.  The more often that happens, the faster conflicts are found and the cheaper they are to fix.  That flies in the face of allowing developers complete asynchronous activity.  Again, periodic sharing is important to maintaining stability.  So, really, the advantage of a distributed system isn't that it IS distributed, it's that it CAN BE distributed.  It allows a more natural period to be established for sharing, but sharing still must take place, and it must be controlled with a consistent policy.  Assuming everyone is working towards the same goal, the next release, the sharing policy must reflect that.

A weekend's worth of successes

I managed to get a number of problems solved this weekend, mostly dealing with the factory model application.  My number one priority was to create a search facility for the factory web application.  I am still using apache Solr as a webservice for the search.

Last Wednesday I tried to create a Vaadin addon that would use a Solr backend to create a search widget.  This was my first real addon, and perhaps I should have tried something simpler as a first attempt.  I did get a lot of experience in debugging the addons.  The basic addon project template wants to use Jetty as the web server for the server side code, which sort of disrupts the normal flow.  In order to run the addon in the GWT shell, you first have to deploy it.  I'm not sure what that buys me, as normally I would use the GWT shell's built-in Tomcat server.  Perhaps at some point I will switch the configuration back to using the built-in Tomcat, but for now I just left it as it is, trying not to change too much in the project it setup for me.

The Vaadin addon was based on GWT's SearchBox and SearchOracle, which I have used successfully in the past for this kind of search.  The SearchOracle has callbacks that are invoked whenever a key is pressed in the search box to provide a list of matches that is refined as the user types.  It uses the Solr webapp directly via GWT's requestBuilder utility class.  The only issue was that the request always returned a status of 0 rather than a status of 200 or some other normal HTTP code.  Googling that situation only lead to sites that indicate that status 0 is a reasonable code only if the protocol is something other than http or https.  I could see the query in the Solr log, so I was certain that 1) the request was http 2) the request was making it to the server and 3) the server was returning 200.  My working theory is that Vaadin somehow messes with the workings of the requestBuilder, which relys on ajax calls to fulfill the request.  I posted a question on the  Vaadin forum but have yet to receive a response.  A new theory that I'm just now realizing is that SOP (same origin policy) is interfering, since the Solr service is running on a different port than the Vaadin test server. I will follow up on that after I post.

Taking a different approach on Saturday, I tried creating a search widget using standard Vaadin components, and having the search running on the server side.  The disadvantage of that is that there is an extra http call introduced per keystroke.  Since Vaadin doesn't have anything similar to the SearchBox/SearchOracle, I would have to create a composite widget that would combine a text field and a list box.  The only hard part is that I need the list box to pop up as soon as the user started typing, and disappear when the user selected something.  I toggled the visible flag to make it appear/disappear, but it would displace the rest of the form, and I needed it to pop up over the rest of the form. Vaadin doesn't seem to provide a simple pop up, only one that's formatted as a window, with a title bar and other unwanted formatting.  To do what I wanted and work within the Vaadin framework, I need to use CSS.

It was time to bite the bullet and introduce 'themes' to my application.  Themes is the Vaadin way of adding CSS and images to a webapp.  If you don't have a theme, a standard one is used without any intervention.  Adding a theme requires that you pull the standard theme out of the Vaadin jar, and put it into your project, so that your new theme can derive from the standard theme.  Then you set your application to use the theme with the setTheme method.

Themes go in a specific folder 'VAADIN/themes' and contain at least one stylesheet 'styles.css' (don't forget to make it plural, I had some trouble when Vaadin couldn't find it because it was named style.css).  To add styles to a component you use the addStyleName method which will add your style to the list of styles.  I created styles for the composite box, the text field, the list and a button.  I also tried to include a label for the text field.  That was one of my first issues, Vaadin always puts labels above the text field, and I wanted it next to the field.  Using the Chrome developers tools I was able to inspect the DOM and found that Vaadin wraps almost everything in <div> tags, which stack by default.  After being unable to figure out how to get the label in a <span> tag, I discovered that CSS now can override the <div> tag's stacking behavior with the 'inline' display attribute.  When they still stacked after setting the tags to be inline, I realized that the enclosing space wasn't large enough, so I had to set that.  Then on to trying to get the pop up list to not effect the rest of the form.  That is done by setting the positioning to be absolute, and positioning it relative to the enclosing box.  After a few hours of trial and error, I finally managed to get something halfway presentable.

Now I can go on a tangent about what a mess the state of CSS is in.  If there's a reason that good web programmers are in such demand, it's because of the steep learning curve to overcome in using HTML and CSS to create decent web pages.  The WYSIWIG designers produce horrendous code that is all but in-decipherable and almost completely unmaintainable, so unless you're working on a one-off project, steer clear. The reason is it's incredibly difficult to translate a reasonable graphical layout to consistent CSS/HTML, while overcoming all the browser quirks.  It still requires the time of a skilled web programmer to create a rational structure that can be understood and maintained.  So I continue to increase my skills in CSS because there's no way out of it. Even with GWT or Vaadin, there's no good way to program yourself out of dealing with CSS.

Saturday I finished getting the search box to display a basic query, and left it at that.  There still was a process issue, my stack now has all these components, web services, database, Solr search service.  I need some way of coordinating all those components.  Additionally, the Solr search service wasn't configured correctly, so it was only returning a subset of items.

This morning, I focused on those two issues.  First of all the Solr search service.  A wide open search was only returning a small subset of what it was supposed to.  Looking at the subset, I could tell that they were only indexing stockrooms and shops, which are both derived from 'internal organization'. That happened to be the last entity described in the DataImportHandler database config xml.  The problem was that it was using the id field as a unique key across all types.  At least with the MySQL implementation, and the standard DDL generated by JPA, the id was not unique across types, so the part table had id's 1, 2, 3... and the internalorganization table had id's 1, 2, 3... as well.  I had to create a key that was unique across the whole database.  The solution was to create a special field in the select: "concat(id,'-type') idtype"  substituting the actual type for type in '-type'.  That created a field called idtype which I could use for all entities.

The second issue I solved was one of process. I needed a process that would encapsulate creating the database, loading data, deploying the webservice API and the seach service.  I have been evolving my thinking on Maven as a build tool.  While it still is the only tool I would consider for putting together 'things', it is sorely deficient on encapsulating 'actions'.  Each POM is focused on creating a 'thing'.  When you need it to perform some action that is above an beyond what is needed for creating a thing, you are cutting against the grain.  So I'm back to my old friend Ant to perform the actions that I need done once all my artifacts are created.  I had an Ant script that I used to startup the search service, that seemed like a good place to start.  There are a set of Maven Ant tasks that allow you to use artifacts in a Maven repository in a fileset and a classpath.  I used both, as I used the classpath generated by my ERPLoader module to run the database loader step, and I used the fileset of the ERPWebservice module to copy the war file to the Solr instance of Jetty.  Now my process was performed with a single ant build.

As my thinking has evolved with Maven, I have been looking at other build tools, most notably Buildr.  There has been a lot of enthusiasm with Buildr, especially it's simplicity compared with Ant and Maven.  But I have a real problem with having to learn yet another expression language (Ruby) just to do what I'm already comfortable with.  I suppose it makes sense for a Ruby programmer to use Buildr, especially for simple cases.  However, in addition to the problem I have with it being in Ruby, there are two other issues that that make me favor Maven in most cases and Ant in the remaining cases.  Firstly, Ruby may be simpler for people to read, but it's not simpler for computers.  Secondly, (and Ant suffers from this too) it is procedural, rather than descriptive, so it's much harder for third party products to infer anything from them.  These two things mean that all the Maven adaptive applications such as Netbeans and Hudson, are less able to adapt to Buildr builds and Ant builds.  Things such as dependency maintenance are a breeze using Netbeans, using point-and-click tools to find and configure my dependencies.  In this respect, Ant and Buildr are too flexible for their own good, while Maven with it's admittedly rigid structure shines.  I think the trick to using Maven and not falling into the hole of mile long pom files, is to separate out actions that are intended to build artifacts with actions that are intended to use artifacts.  Use Maven to build artifacts, but use Ant or some other scripting language when you're in use mode, since the act of building tends to be pretty straight forward, but the act of using is much more open-ended, and requires a lot more flexibility.

3D Graphics in a browser and a reconciliation between JPA and JAXB

After the week's success with Vaadin, I did some research into running 3D Graphics in a browser, and got around to adding XML binding to my factory model this weekend.  That in addition to some gardening Saturday afternoon, my final choir concert Saturday evening and going to see the Seattle Symphony this afternoon.

A few months ago on the Google code blog, they demonstrated a port of Quake II to GWT, and had it running in a browser.  That's right, a 3D real time game running in a browser without any plugins (not even Flash).  HTML5 which is semi-supported by Chrome, Safari, and Firefox includes WebGL which is based on OpenGL. WebGL is a canvas based drawing/rendering system that exposes a JavaScript API for creating 2D and 3D graphics.  I had some interest in this, as I still have parts of a 3D networked multiplayer that could make use of this.

My scope for my research was pretty simple: could I create a very simple Maven/GWT project that would create a simple 3D webapp.  I started with a basic Maven/GWT webapp project from an archetype  I was able to follow the instruction, although I added the step of temporarily removing my local repository mirror since I was dealing with a foreign repository, and I didn't want the mirror to interfere.  From there I went to the GwtGL site, which is basic GWT bindings for the WebGL interface.  It had two section, how to add what you need to your webapp project to use GwtGL and a sample web app that just displays a white triangle in a black square.

Adding GwtGL was pretty straight forward.  Add a div tag to be your canvas, inherit the library in the GWT project descriptor and include the library (and source)  in your classpath.  Before I could do anything, I needed to download the GwtGL source and binaries and install it in my local repository, as it's not on Maven Central yet.  I'm using Sonatype Nexus as a repository manager, and it has a nice artifact upload interface.  The standard setup for Nexus includes a 3rd party repository for uploading 3rd party artifacts that have yet to make it to Maven Central.  In the zip file download from GwtGL is the binary, source and javadoc jar plus the POM file, and I uploaded all of them to Nexus.  For some reason the groupId is com.cooglecode, I'm not sure what that's about.  Then I added to the main.html the div tag:

<div id="gwtGL"'></div>

The inheritance tag goes in the GWT project descriptor:


  <inherits name='com.googlecode.gwtgl.binding'/>


And finally the binary and source jars need to be added to the dependencies:


        <dependency>
            <groupId>com.cooglecode.gwtgl</groupId>
            <artifactId>gwtgl</artifactId>
            <version>0.3</version>
        </dependency>
        <dependency>
            <groupId>com.cooglecode.gwtgl</groupId>
            <artifactId>gwtgl</artifactId>
            <classifier>sources</classifier>
            <version>0.3</version>
        </dependency>

The sample code had one issue, it would appear that a class name has changed, but the sample code wasn't updated.  After a bit of poking around, I discovered the updated class name for WebGLFloatArray was Float32Array.  A quick compile, and run and I had a browser with a black square with a white triangle.  Maybe next weekend I'll look at what it will take to convert the 3D game client to run in a browser.

Today I spent some time making an export and import API for the static objects in my factory model.  I wanted to be able to import and export to XML all of the parts and build steps, shops and stockrooms that comprise what I call the 'factory matrix'.  This is everything that has a static description which isn't supposed to change, and so would probably loaded once into a database and rarely changed.  In order to be able to do this with JAXB, I needed to break any cyclic links or bi-directional references.  I decided to use the parent/child reference technique where I set the parent reference to transient with the XMLTransient annotation, so it's not set in the initial traversal, but set later on with a callback called afterUnmarshal.  The only issue I had was that the XMLTransient annotation needs to go on the bean getter, not on the field itself.  I had it on the field, and was still getting cyclic reference exceptions when marshaling the factory matrix.  Putting it on the getter, and I was back to 'build success' which always makes my heart happy.  In addition to the import and export function, I added a lot of unit tests, hopefully my code coverage reports won't look so dismal tomorrow after new reports are generated.

An Aha! coding moment

The Google Web Toolkit (GWT) has held the promise of web-programming nirvana since it's introduction a few years back.  Yet in practice, it has fallen short of it's heavenly goals for several reasons.  First of all, it adds minutes to a code-compile-test cycle that coders have gotten used to taking seconds.  Debugging in the GWT shell requires the patience of a saint.  And lastly, it forces an awkward split between the code that runs on the browser and the code that runs on the server.

Admittedly, most of these problems and more exist in traditional web-programming environments, which is why I've always considered the browser-based apps as a huge step backwards in user interfaces, both in terms of coding them and using them.  It also means that anyone with the wherewith-all to do web coding are the stars of the coding world.  Everything is moving to the web, from games to business app, and if you're not on the band-wagon, your going to be the COBOL coders of the modern era.

GWT was supposed to solve all that by allowing all web coding in Java, there-by allowing the great abundance of Java tools to be used, and anyone with Java skills to become a web-programming star.  It was genuinely easy to create the ubiquitous 'hello world' web app and have it up on your browser in seconds.  The problem is that it didn't scale.  The minute you tried to do anything real with it, the compile time started taking forever.  You couldn't split up your code and have some of it pre-compiled, everything that ran on the browser side had to be compiled in one step.  Add to that the lack of libraries and features that Java programmers have gotten used to over the years.  You couldn't use one of the many XML libraries available, you were stuck with a very basic XML parser.  There wasn't a tradition 'Class' object so no introspection, or dynamic object creating, which is the main-stay of anyone that is trying to separate out application code from architectural code.  One way of over-coming much of this was to move code over to the server side which had access to the complete richness of the Java language and libraries.  In all MVC architectures that I've seen thus far, this was done by splitting the model side and providing a synchronization layer between the client and the server.

Then, last week I was reading on the Archiva developers mail list about a planned refactoring, which would include the UI.  It was suggested to use GWT for the UI, but Vaadin was also mentioned as a possibility.  I had not heard of Vaadin before, so I did a quick google search for it and looked through some sample code that they had.  It looked a lot like a GWT app, a module that was loaded at the start, creating a UI with java classes, a simple model binding for the UI.  I was amazed when I downloaded it and compile it, it compiled in seconds, with only the standard Java compiler, and yet it ran like a full browser app.  Where was the GWT compiler?  How was all this UI code running in the server?  There was no messy configuration, no limitations on the Java code.  It was like compiling a simple Swing program, but it ran in the browser, just like a GWT program.

I had to find out what the magic was behind all this.  After digging around a bit, I discovered that they had made a fundamental shift in the MVC architecture.  Instead of splitting the model and keeping it synchronized, they split the client.  Aha! all of a sudden, the possibilities became apparent.   The Vaadin library had come from a different place, by creating an abstract terminal model for the browser prior to the advent of GWT.  When GWT came along they adopted it into their model.  Now we could have a GWT app, with most of the programming done in the server.  The only time you needed to deal with GWT is when creating custom widgets.

I wanted to put Vaadin to use, and I had been working on my factory model application during my spare time for several week prior (as I mentioned in my last post on JAXB/JPA).  I decided to created a GUI front-end for that model with Vaadin.  In almost no time, I adapted their form example to my part object, so now I had a simple CRUD (Create-Read-Update-Delete) form for parts.  Then, I adapted that form for stockroom, with equal ease.  I factored out a basic CRUD form, and then created forms for all of the static objects in the factory model.  I added drop-down lists for all one-to-n relationships that had a reasonably small n-count, and the dual-list chooser for n-to-n relationships.  Basically, I had created a substantial fraction of an entire business application in just a few spare hours over a few days. The great part was that it worked without coercion directly with all my JPA objects.  Here was a library that 'just works' with JPA.  Most of the nuances of servlet programming were invisible to me (everything ran inside a session).  And I had none of the issues of dealing with the browser.

The Aha moment wasn't actually mine, it was a 'domino' Aha moment.  Obviously the people at Vaadin (which is a Finnish word for a semi-domesticated adult female reindeer) had the initial moment when they had the fore-sight to tame the web side by creating the abstract terminal model.  They then embraced GWT as the language of choice for creating the web side of the terminal model.  For me, the Aha moment was the sudden realization of all the possibilities that this confluence enabled.

I will continue to work with this library to see how well it does for other typical web application stumbling blocks such as grids for large tables, and editable grids.  It has been a pleasure to work with so-far, hopefully it scales well, and is easily adaptable. If it continues to make coding fun, it will certainly become my primary means of coding for the web.  I think that basic GWT still has a place, especially for browser applications that don't conform to the terminal model.

Reconciling JPA with JAX-B

I've recently begun work on refining the manufacturing plant model that I had created a year or so ago.  Part of the work was to split up the object model into sub categories, and prevent cyclic dependencies between them.  I came up with three categories so far: static entities that are descriptions of physical real-world entities, such as parts, and shops, and vendors; inventory tracking records that keep track of all parts currently in the system, and historical records of parts no longer in the system; and various business documents that keep track of everything.


One of the problems I faced as I split out these categories, is all the bi-directional relationships defined.  When I first started to define the model, I basically made all relationships bi-directional figuring I would pare it down in the future as needed.  But all those bi-directional relationships create cyclic dependencies throughout the system, which impedes breaking things into modules that cannot have cyclic external dependencies, and problems if I was to use JAX-B to export the objects as XML.


For the most part changing bi-directional relationships in JPA is straight-forward, simply remove the list from the 'n' side of the relationship, and provide a query to replace it.  So for example in my object model, Stockroom used to contain a list of RealPartSet objects called inventory.  That has several problems, firstly anytime you need to get a Stockroom object, you drag it's list of inventory along with it.  That's not a big deal in small systems, but it certainly doesn't scale well.  The second problem, is that Stockroom is in the static entity module, and RealPartSet is in the part tracking module, so there cannot be a bi-directional relationship.  So I removed the list of RealPartSet objects from the Stockroom, created a new object called Inventory which contains a list of RealPartSet objects, and added a query to the Inventory object, findByStockroom.  Then anyplace I used to call Stockroom.getInventory(), now I call Inventory.findByStockroom(stockroom) to get the list.


One part that was a little more tricky, was in bi-directional many to many relationships.  For instance, a part may be source from several vendors, and each vendor supplies many different parts.  That is a real-world relationship that I could not dodge.  I could set up a cross-reference table, but my main goal in this project is to figure out how to best use JPA when modeling real-world problems.  So I continue to use the ManyToMany annotation, but I removed one side, in this instance, the vendor no longer has a list of the parts it supplies (which would have introduced a scaling problem anyway).  Now the problem of forming a query to fetch the list of parts supplied by a single vendor.  I looked it up in my books, and tried googling it, but found no results that specified exactly the JQL query that I would need.  Doing a more refined search brought me to  the code ranch which after reading down a bit suggested a query similar to this:


select p from Part p join p.vendors v where v.vendor = :vendor 

However that did not work it complained about v.vendor.  So I just tried a very simple approach with this query:

select p from Part p join p.vendors v where v = :vendor

That did the trick.  Now that I figured out the pattern, I applied it to many of the many-to-many relationships in the object model.

One final issue, using JAX-B bindings with JPA objects.  I would like to be able to use the JPA objects directly in webservices that use JAX-B to convert the Java objects to XML.  The problem is that JAX-B does not handle cyclic relationships well.  They have a number of solutions to creating XML from cyclic java objects, the first being to make the parent reference in the child object transient and filling it in later.  That would work well, if there is a single defined parent child relationship AND the child object never needs to be transmitted by itself.  The second approach uses XML id attributes to reference objects, but the problem is the id has to be unique in the whole document, and anything I would use as an ID is only unique for the class of objects.  There is a third way, some kind of interface that you can supply an implementation to resolve cycles 'on the fly', but the link provided in the JAX-B site is broke.  So I'll have to do some more research on that.

So I don't have an answer on how exactly I'm going to provide JAX-B bindings consistently on all my JPA objects.  I will have to research further and see if I can't figure out a good way to handle that without having to resort to the data-transfer-object anti-pattern.

My One Byte of Memory

Now that everyone has billions of bytes of memory on thumb drives and SD cards cluttering their drawers, I made a project to create one byte of memory.  Actually it's only a half a byte that can be accessed.

It started last weekend, when I was looking for a project to waste my time on for my three day weekend.  I went off to Borders, and found a copy of Make: which featured the new Arduino microcontroller board and development software.  After reading about all the things I could accomplish with it, I went to Fry's electronics to get the parts necessary to build my own Arduino board with a breadboard and a Amtel Mega microcontroller.  I'd been looking to do this for a while now, ever since Tracy's son got me a 'game of life' circuit kit for Christmas which used an Amtel microcontorller.  I've been half heartedly looking around for development rigs to be able to program it.  Now, this magazine had everything I needed down the parts list.

First issue though, Fry's only carries Parallex microcontrollers, and they were rather expensive.  Also they didn't have the USB breakout board or the 16 Mz crystal.  Not to be daunted, I got the rest of the stuff I needed, and a few various digital IC chips to play with, just to get used to bread boarding again.  They cost less than a dollar each, and I wanted to use them as guinea pigs on my rusty circuit building skills.

One of the digital IC chips I got was a 74HC273 chip which is an octal latch.  Which is a fancy way of saying one byte of memory.  So my first project was to hook it up to something.  I got out my breadboard, a 6 switch dip switch, a 74LS02 quad OR gate to use as a buffer, and 4 leds, plus various resistors and jumpers.  Of the 6 switches, I hooked one up to the clock, one to the reset and 4 to the inputs of the latches.  I hooked the output of the latches up to the buffer, and the buffer to the leds.  Of course nothing happened when I turned it on.  After a little bit of research, I was reminded that CMOS chips like the 74HC273 need to have a resistor to ground on the input, so that when the switch is open, the input voltage has someplace to drain.  I added in a few resistors, and viola! I have a half a byte of functioning memory.  I can set the four input switches to whatever settings, toggle the clock switch, and the LEDs light up based on which of the input switches were on.  And they remain that way, even if you change the input switches.  So it 'remembers' the switch states until you throw the clock toggle again or switch on the reset (or unplug it).  Here's a picture of the final product:

With this I can remember a number from 0-15 as long as I keep it plugged in.  It may not seem like much, but put a billion of these things together, and now you're talking.

I have since gotten the microcontroller and the crystal needed for the main project I wanted to work on.  Hopefully I'll get that USB breakout board soon, and I'll be able to program my new microcontroller.  Then I'll have something real to report on.  Until then I can bask in the dim LED light of my amazing grown-up tinker toy project.

Block Math Conclusion

Last week I left the definition of 'midway' hanging.  To define midway, I'm going to use my old friends 'imaginary blocks'.  Let's say you have a chipped-stack, that is all rows have the same amount of blocks except the top row.  We can fill in the top row with imaginary blocks as so:

XXYYY
XXXXX
XXXXX

Here the 'Y' blocks are imaginary, we've used our imagination to fill in the top row.  Now we compare the amount of X blocks in the top row with the amount of Y blocks by stacking them:

YYY
XX

It would appear that the X block amount is less than the Y block amount.  My definition of 'midway', is when the X block amount is the same size as the Y block amount.  If the X block amount is greater than the Y block amount, we can say the top row is greater than midway.  Similarly if the X block amount is less than the Y block amount, we can say that the top row is less than midway.

So why is midway useful?  In the real world, when we need a clean stack, one isn't always available.  In fact, for stacks with a large amount of blocks in each row, it is easy to imagine that one is rarely available given an arbitrarily sized block-bag we are trying to arrange.  By comparing the real blocks with the imaginary blocks on the top row, we can say that the top row is more real than imaginary and consider it whole, and we can say if the top row is more imaginary than real, we can disregard it.  Additionally, assuming that the top row will have one amount with equal probability to any other amount, on average the number of times that we consider the top row by filling in the blanks will be equal to the number of times we disregard it, which allows things to even out in the end.

Bear with me for one more scenario, and then I will summarize and get off my soap box.  Imagine that Costco has a sale on garden boxes.  The garden boxes come in many sizes, but all of them are square in shape, that is each side is of equal amount.  You got a bag full of corn seeds from your Aunt for Christmas.  You need to figure out what size garden box to buy so that you can plant all your corn.  Since each seed needs to be planted some distance from the other, you devise a scheme where you can use some of your blocks to represent where you are planting your corn seeds in an imaginary arrangement, whereby each block represents a size equal to the distance needed between seeds.  So you find a block bag with an equal amount of blocks as the amount of corn seeds in your bag.  Now you need an arrangement strategy for finding the smallest 'square-stack' needed for all the blocks in your bag.  One strategy might be to arrange in a 'spiral' pattern, always trying to maintain the 'squarest' shape.  As you pull each block out of the bag, here's the arrangement:

X

XX

   X
XX

XX
XX

XXX
   XX

XXX
XXX

XXX
XXX
X

XXX
XXX
XX

XXX
XXX
XXX

You continue this process until you run out of blocks.  Now you can remove the blocks in the longest side to represent the amount of 'corn-seed-distance' units needed to allow you to plant all your corn.  You then make a stick that represents the 'corn-seed-distance' unit, take it down to Costco, and find the garden box that will allow that stick to be placed end-to-end the correct amount of times, and purchase that size of garden box (hopefully it will fit in your car).

So what have I done here?  Not much, really, most of the problems I've solved with these operations seem trivial.  Some of the scenarios are rather arbitrary (how often does Costco have an assortment of square garden boxes of different sizes on sale?) What I've done is to think about the processes involved and come up with a new simple vocabulary to describe them.  That allows me to think more clearly about the processes, and not be saddled with my prior knowledge sneaking in and providing short-cuts that may not be intuitive to other people.  It allows me to communicate better with other people that don't share my knowledge and, I hope, makes it easier for other people to understand.  In short it makes me a better teacher.

Much of the vocabulary used in today's math is foreign to the rest of our language, and so words like 'roots' and 'fractions' become scary unknowns to most individuals.  The arcane symbology used to describe all but the simplest formula is enough to send most people running. Much of what we are taught of math is from one perspective: sterile abstract operations such as addition, subtraction, multiplication, and division.  Memorizing multiplication tables, and performing long division.  Which makes us into mistake-prone calculator, and a little less valuable then the cheap electronic calculators you can find at the dollar store.  We should be teaching by saying first, heres's a problem, (pause to let the problem sink in) and second, here's a solution and how it works.  Instead we are teaching by saying: here's a solution, and oh-by-the-way, there are problems out there that might be solved with it, trust me.

Playing with blocks, making arrangements, talking about types of arrangements, and how they can be used to solve problems seems to me to be a simple and intuitive way of teaching the fundamentals of mathematics.  In the past three posts I have described addition, subtraction, multiplication, division, ratios, roots, negative amounts, and rounding, without ever using a single number and with a limited vocabulary of common words.  The blocks allow us to think abstractly with representations, and to visualize these process as they are occurring.  If, when teaching math, we lay that as a foundation, we can then teach the tradition constructs and formulas of math and have something concrete to relate it to.

For those of you with only the tradition constructs and formulas of math in your knowledge set, and without the benefit of higher math which forces you to think about the processes involved, it may be hard to reconcile block math with real math.  So here's a cheat sheet.  The scenerio of the block bank robbers teaches about division.  You have some amount of blocks, and you need to divide them among your gang.  There's even a remainder (remember those?).  The scenerio of the siblings buying a new game and pooling their money teaches about addition, subtraction (or negative numbers) and comparison.  You need to add the amount of blocks, compare their amount with the imaginary amount (a negative number), and subtract the imaginary amount.  The scenario of paying the waiter a tip teaches multiplication, ratios (fractions), and rounding. Building the clean-stacks is dividing and multiplying: the first stack is the bill divided by some number, and the second stack is the height of the first stack multiplied by some other number.  Building a pair of clean stacks with equal amount of stacks, but unequal amount of blocks in each row is a ratio, and evening out a chipped stack to a clean stack is rounding.  And finally the square garden box scenario teaches about square roots (appropriate to talk about gardening when discussing roots).  The size of the box you need is the square root of the amount of corn seeds that you have, rounded up to a whole number.

And lastly for you mathematics fans out there, I have nothing against your specialized vocabulary and arcane symbology.  Unambiguous definitions and rigorous proofs for the most part require them.  But when trying to communicate with someone that doesn't understand the vocabulary, it might be easier to explain things using a common vocabulary even if you're not being as precise as you might be if you were communicating with one of your own kind.

Block math can be used to describe succinctly operations that involve integral things.  A thing is integral if it can't be cut in half and still be useful.  You can't cut a computer in half and have two half computers.  You can't cut a penny in half and have a monetary amount of half a cent.  You will have two pieces of junk.  It can't be used to fully describe things that can be cut in half.  A yard of cloth can be cut in half to make two half-yards of cloth.  These are very distinct ideas, and yet our temptation is to use one concept 'numbers' to represent both ideas.  Block math avoids some of the pitfalls by making clear the distinction.  Perhaps in another series of posts I can talk about the other half of the number problem.  Until then, have fun playing with your blocks!

Block Math Part 2

Last week I talked about a system that consists of blocks, block-bags, and arrangements. I introduced two types of arrangements: block-rows, and row-stacks.  Row-stacks can be considered 'clean' if each row contains the same amount of blocks.  I'll introduce a new word for the inverse condition, row-stacks can be said to be 'jagged' if the rows have different quantities.  And one more word 'chipped' for the condition that all the rows are of equal length, except the top one which has less than the rest.  Here's some examples:

clean:

XX
XX

or

XXX
XXX

jagged:

XXXX
X
XXX

or

XX
XXXX
XXXXX

and chipped:

XX
XXX

or

XXX
XXXXXX
XXXXXX

All chipped stacks are also jagged stacks, but not vice-versa.

I'm going to define some operations in abstract terms so that they are more-or-less unambiguous.  The first operation I will call 'sharing'.  Given a clean stack of blocks with a certain amount in each row, you want to share the blocks into an amount of bags equals to the amount of blocks in each row.  Easy right?  You put one block from each row into each bag.  If the stack is clean, you will have an equal number in each bag (since each row contains an equal amount of blocks, therefore each row will have one to offer each bag).  Given a stack of blocks that is chipped, we can use the same process, and each bag will be 'almost equal', that is the difference in the amount of blocks in each bag will be at most a single block.  We can confirm this intuitively by flipping a jagged stack on it's side and observing that each row now only differs by at most a single block from any other row:

XX
XXXX
XXXX

becomes:

XXX
XXX
XX
XX

So now the problem becomes:  given a bag of blocks, form a clean or chipped stack with a certain amount in each row.  We have imagined a row that we can use for a template.  Now we can pull blocks out of the bag and place each block in a row until the amount of the row is equals to the amount in the imagined template row.  At that point we start a new row on top of the old row and repeat the process.  We continue until there are no more blocks left in the bag.  What we will have at the end will be either a clean or a chipped stack.  I will call this operation 'clean-stacking' because we are trying to build a clean stack.

So in order to share the blocks in one bag fairly between a set of bags, we do a clean-stacking operation, and then we do a sharing operation.  This works as long as we define sharing fairly as being that the difference between the amounts in each bag is never greater than a single block.

A new scenario: A block hunt. We have hidden all the blocks from a block-bag around an area, and now we have a certain number of contestants that will find them.  The contestant with the most blocks gets a blue ribbon, the contestant with the most blocks after that gets a red ribbon, and the contestant with the most blocks after that gets a yellow ribbon.  The contestant with the least amount of blocks gets a black ribbon.  How to determine who gets which ribbon?  First, form a jagged stack arrangement by forming a row from each bag and stacking them.  Next form an 'ordered-stack' by finding the longest row in the first stack and placing it in the second stack, then repeat the procedure until the first stack is empty.  As long as you keep track of who belongs to each row, the bottom row will get the blue ribbon, the next row up will get the red ribbon, and the next row up will get the yellow ribbon.  The row at the top gets the black ribbon.  This type of operation is called 'ordering'.  There are actually many variations of this operation, but the one described above is simple, and works well for small stacks.  What to do in case of a tie?  Actually, it doesn't matter, since we made up the rules of this game we can make up the rules arbitrarily for tie cases, and as long as all contestants agree to the rules, and the rules don't change during the course of the contest, it should be considered fair.

One more arrangement, is actually a pair arrangement.  Given a clean stack of blocks, form a partner clean stack with the same amount of rows.  The amount of blocks in the rows of the two stacks can differ, as long as they are the same in each stack.  For example:

XX   XXXX
XX   XXXX
XX   XXXX

If we add or remove rows from both sides, we can see that the total amount in each stack grows or shrinks relative to each other.  So I will call these stacks relative-stacks.  When would you need to have relative-stacks?  Here's a scenario:  You're dining out, and the waiter is very prompt and friendly.  You know he probably only earns a small salary, and you want to reward him extra for his good service.  How do you determine what is a fair amount?  You could just give him some fixed amount, or you could consider that the amount of effort that he spent is somewhat relative to the size of the bill you receive at the end of the meal.  In order to be fair and reward him relative to his effort, we can make a pair of relative-stacks, the first being the a stack made from blocks equal to the number of dollars in the bill, and the second being a partner in the relative stacks.  The number of rows in each of the partner stacks is some arbitrary amount that has been determined by good-manners experts as producing a fair amount.  So first we do a 'clean-stacking' operation for the first stack based on a block-bag equals to the amount of dollars in the bill, and the we do a 'clean-stacking' operation from our reserve block-bag but stop when the stack amounts of the stacks are equal.  We give the waiter an amount of dollars equal to the amount of blocks in the partner stack.  So for example:

XXXXXXXXXX   X
XXXXXXXXXX   X

The stack to the left represents the dollars in the bill, and the stack to the right represents the dollars in the tip. This way we ensure that for a larger bill, we give a larger tip, since we consider the bill to be relative to the amount of effort the waiter made, and wish to reward him fairly.  The amount in each stack is somewhat arbitrary, except that the bill stack will tend to have many more than the tip stack and the amounts should be consistent across time.

So what happens if when you are doing the 'clean-stacking' operation, you come up with a chipped stack?  You could do a number of things, you could form the partner stack based on the amount of full row stacks, or based on the amount of stacks full or not.  Or we could introduce a new term 'mid-way'  If the amount of block in the top row is greater than mid-way, you would base it on the total amount of all stacks.  If the amount of blocks in the top row is less than mid-way, you would base it on the amount stacks of only full rows.  If the amount of blocks in the top row is exactly mid way, you would have to find some random way of determining what to use.  Next week I will better define the term mid-way and why it might produce a more-fair relative stack.

The Inevitability of Parallel Universes

This NPR article got me thinking: why wouldn't it be true that parallel universes must exist.  Of course the term universe is being used rather loosely, if the universe IS infinite, there couldn't be a parallel universe, only a parallel containment unit within the universe (such as a parallel galaxy).   

After a bit of thinking I realized that the hypothesis is false if you accept the idea from my previous post that the universe is not only infinitely big, but infinitely precise.  The infinite precision would allow for an infinitely sized universe with no requirement that there ever be a 'parallel' containment unit.

There is probably some proof of this idea in set theory somewhere, but as a philosopher, I have immunity from having to come up with proofs.  Maybe if there's a mathematician out there they can supply that.  A good place to start might be this wikipedia article

Rationalizing Free Will and Determinism

On the face of it, free will and determinism seem as irreconcilable as science and religion.  With determinism, everything is pre-ordained, everything happens according to some master plan.  If that is true, how can we have free will, we could just blame all our actions on this master plan, and we are just helpless bits of silt being washed down a stream.  And yet, I can't help but believe in free will of some sort, I seem to be in control of many of my actions.  When it comes time to assign blame for my actions, the buck stops here.

I think the key is predictability.  With determinism, we tend to infer predictability.  In the scientific model, once we can determine the master plan, or some piece of it, we can predict the future.  We figure out the master plan for planetary orbits, and now we can predict when certain celestial events will occur.  There still is an inference, as when we describe the master plan, we describe it with a mathematical model, so our inference moves away from prediction based on previous occurrences, to the inferences that our description matches reality.  Basically, the scientific model relies on determinism, and the ability to isolate a certain piece of the master plan that is uninfluenced by other pieces.

On the nature of reality, I think we can ascribe two attributes to the universe.  One, that it is infinitely large, and the other that it is infinitely precise.  By infinitely large, that is to say it has no bounds, no matter what level you look at the universe, you can always go one level out in containment.  Planets are part of star systems. star systems are part of galaxies, galaxies are part of something else.  We use the term universe to describe not some final containment category, but to define a system of all containment categories, which I tend to believe is infinite. The other attribute of the universe is that it is infinitely precise.  By that I mean that we can go in the reverse direction, and assign containment categories to an infinitely smaller and smaller set.  Each level in minuteness we discover leads us to believe there is one level smaller, and this probably also goes on forever as well.

Now to problems with predictability.  Let us think of a pole with a magnet near the base of it.  Tethered to the top of the pole by a string almost as long as the pole is a magnet of the same polarity, so that it would naturally be repelled by the magnet at the base of the pole, which it would otherwise come into contact with.  Lift up the tethered magnet and let it free fall.  It will avoid the pole and be diverted to one side or the other.  If the apparatus is built correctly, we cannot predict to which side the tethered magnet will divert, and yet it is a very simple mechanism easily described in this paragraph, and easily constructed.

So why can't we predict the above experiment?  Why doesn't it fit into our system of using mathematical models to calculate outcomes?  I believe that it does, as it is fundamentally no different than any other aspect of reality.  It is in-calculable because of the infinitely precise nature of reality.  To model a reality of infinite precision, you need a mathematical model of infinite size.

If it's that case that all of reality is infinite in precision, how can we model any of it with finitely sized mathematical models?  We can because the aspects of reality that we model conform to describable patterns, so that we can describe not the actual model, but can describe the basic shape of the model.  We can use advanced mathematics to calculate the outcome of certain describable patterns, even though we cannot express the actual model due to it's infinite size. The problem with the above experiment, is that it defies this reduction, and so must be described by a model of infinite size, which would be impossible to express, much less calculate.

So that is how we have a system that is both deterministic, and impossible to predict.  One might be tempted to state that determinism is simply the ability to determine, or predict, and so say that this denies determinism. But I believe that determinism should be defined, not by our ability to predict, but on whether, given infinite capacity, something would be predictable, and to that I would answer yes.  For the universe to not be deterministic, it would have to be influenced by some external force, and yet you could easily wrap that external force into the definition of the universe and it becomes deterministic again.  For the universe to not be deterministic, it would have to make choices, and I don't believe it is capable of that.

To help differentiate that determinism does not mean OUR ability to predict, but simply predictability given infinite resources, I will use the term 'virtual determinism'  By that I mean that something is deterministic and yet impossible to predict given finite resources.  I believe that I have described above why the infinite nature of reality can cause things to be unpredictable, but there is another related aspect of reality that has the same effect, that is the infinite inter-connection of reality.  In my second paragraph I added the qualification of using the scientific method as that an aspect of reality must be able to be isolated from all other aspects, and yet that is also impossible based on my assertion that reality has infinite inter-connections.  However, some aspects can be isolated relatively well, such that the error caused by outside influence is negligible.  That leads to a further impediment for prediction in a deterministic world.

I also believe this unpredictability is unrelated to Heisenberg's Uncertainty Principle, which I might generalize as 'measurement or understanding at one level of reality, requires elements from the level of reality contained in (at least one level lower than) the level being measured'.  So to precisely measure elements at the atomic level, we need first to understand and master elements at the subatomic level or lower, which we have not done yet.  Once we have a better understanding and control of subatomic matter, we will fill in many of the blanks and paradoxes that haunt quantum mechanics.  But we will still have some 'Uncertainty Principle' in regards to subatomic matter until we go down yet another level in our understanding and mastery.  I believe this can be seen historically, as our mastery of the atomic level filled in many blanks and paradoxes in our understanding of our reality.

So given a belief in determinism (or virtual determinism), how can I believe also in free will?  Basically, I don't.  What I do believe in I will term 'virtual free will'.  It's an illusion of free will based on our inability to predict a deterministic reality.  If someone says "I will raise my right arm", we cannot predict that they actually will, without the leap of faith of taking them at their word.  We cannot build a mathematical model that connects the path from the statement to the action.  We can make a judgement based on past experience that they will follow through on their word, but that is all.  As conscience beings we have inside information that can better predict our actions than the outside world can using a scientific model and the more information about our future actions we divulge, the better the outside world can predict.  But no one (including ourselves) can make a scientific statement about our future actions.  We do not yet have the understanding to connect our motivations to our actions, other than to infer that one tends to cause the other.  This, of course, includes our motivation to demonstrate our 'free will' whenever the need arises.  We make a decision about which action to take, but that decision is at once deterministic, AND unpredictable with a mathematical model.  We don't 'cause' the decision to be made, but we have enough inside information about it in the moments leading up to it to believe we are the cause because we have thought about consequences which feed into the deterministic decision making process.  The thought of the consequences of our actions along with our character and willingness to face consequences (or ignore them in the decision making process) are all parts of some vast and possibly unknowable deterministic equation.

Because one's character (or tendencies, innate and learned) are part of the equation, in a social context we can make a judgement and take action based on someone's actions even if we don't fully understand the processes involved in the decision making.  The actions can be to help alter a person's learned tendencies, by punishing unwanted behavior, or rewarding desired behavior, adding the punishment and reward to future consideration of consequences.  They can be to isolate or reduce the power of someone with unwanted innate tendencies, or give greater power to someone with desired innate or learned tendencies.  We have to try to make a distinction between innate and learned tendencies, even though we don't understand fully the distinction, and don't even understand the process that turns tendencies into actions.  So when I say 'virtual free will' I don't mean to confirm free will or deny it.  I simply mean that for all intents and purposes we can consider free will to be real, and part of the reality that we live in.

For many millennium  man has drawn the conclusion that the sun will rise in the east, and yet only when we understood planetary orbits did we have scientific confirmation that this conclusion was true. That did not mean that the process was not in place until we understood it.  Prior to our understanding, we ascribed a will to the sun to rise in the east, until we could see how it fit into a coherent description of reality.  Such is the will that we ascribe to ourselves.