[werkflow-user] BPEL support

[Paul Russell]

Hi all,

I was pondering trying to start up an open source project (probably
apache licensed) to build an open source implementation of a BPEL
engine, and tools. A friend of mine kindly pointed me towards Werkflow
as something playing in this space. I have to say, it takes an approach
I've not seen before, which is refreshing: I come from a websphere
process choreographer background, which works using database-based
persistence to store state, and JMS + Message Driven Beans to provide
activity scheduling and dispatch - I was kind of /assuming/ a similar
architecture for implementing BPEL.

I'm currently trying to understand the Werkflow architecture in this
context. Process Choreographer implicitly supports clustering, for
example, because of its use of messaging to dispatch activities. I'm
trying to get to the bottom of how Werkflow might achieve the same
thing.

I guess what I'm trying to assess is how far off being production ready
for werkflow is, both in a generic sense and in a BPEL sense. So for
example, for me to consider this production ready, I'd expect it to:

* Support running in a multi-node environment
* At least with High Availability.
* Preferably with load balancing and high availability.
* Support Compensation (although I /think/ this could be achieved by
including it in the appropriate personality.
* Support graphical debugging through development tooling.
* Support development tooling ;)

Do you guys have any thoughts as to whether these are things you were
expecting to support as part of werkflow? If so, do you have a road map
to get there? I'm quite willing to help if you can point me in the
right direction!

Thanks in advance,


Paul

[Brian Topping]

Hi Paul,

I'm in the same boat with you, but I've gone a bit further down the path
understanding the system. I'll take a stab at answering your questions,
hopefully someone can correct me where I get it wrong.

Post by Paul Russell
I was pondering trying to start up an open source project (probably
apache licensed) to build an open source implementation of a BPEL
engine, and tools. A friend of mine kindly pointed me towards
Werkflow
as something playing in this space. I have to say, it takes
an approach
I've not seen before, which is refreshing: I come from a websphere
process choreographer background, which works using database-based
persistence to store state, and JMS + Message Driven Beans to provide
activity scheduling and dispatch - I was kind of /assuming/ a similar
architecture for implementing BPEL.

Wrapping Werkflow with this functionality should not be difficult.
Repository interfaces exist inside the system, so it's really just a matter
of extending your persistence mechanism to include them on the back end.
Adding the MDB transport on the front end should be a couple dozen lines of
code.

Post by Paul Russell
I'm currently trying to understand the Werkflow architecture in this
context. Process Choreographer implicitly supports clustering, for
example, because of its use of messaging to dispatch activities. I'm
trying to get to the bottom of how Werkflow might achieve the same
thing.

If things were wrapped in MDB, you are correct that clustering would be taken
care of.

Post by Paul Russell
I guess what I'm trying to assess is how far off being
production ready
for werkflow is, both in a generic sense and in a BPEL sense. So for
* Support running in a multi-node environment
* At least with High Availability.
* Preferably with load balancing and high availability.

This probably depends more on your container than Werkflow. If you had a set
of routines that you wanted to do this, how would you do it? Werkflow just
provides the entry points for you to call. It's very independent of
container, so you don't have to worry about all the headaches of the engine
playing with the container state.

Post by Paul Russell
* Support Compensation (although I /think/ this could be achieved by
including it in the appropriate personality.

Not sure...

Post by Paul Russell
* Support graphical debugging through development tooling.

This is one that I am particularly interested in myself. It's not there yet.
I'm looking at generating my source files from XMI, and I could imagine that
there would be a way to have a fat client that could graphically trace the
UML that was in the XMI. I have some ideas here, but I don't really
understand what ought to happen because I've never seen a commercial workflow
engine in operation.

Post by Paul Russell
* Support development tooling ;)

Would generation from UML work?

Post by Paul Russell
Do you guys have any thoughts as to whether these are things you were
expecting to support as part of werkflow? If so, do you have
a road map
to get there? I'm quite willing to help if you can point me in the
right direction!

I can't speak for the rest of the team, but I am working on these as a part
of my project. It's just a matter of time. But that still doesn't answer
your question. Do you have the resources to help with this or are you
needing something that is more complete from the outset?

I'm personally not that impressed by the other solutions out there, and feel
like I'd rather invest in Werkflow than try to fix what's broken in the
backend on the other tools.

If/when I attack this, it will probably be as I described above. It would be
great to have you help with rough text or graphic descriptions of what you
are looking for. Can you provide some ideas?

-b

Post by Paul Russell
Thanks in advance,
Paul
--
Paul Russell
_______________________________________________
werkflow-user mailing list
http://lists.codehaus.org/mailman/listinfo/werkflow-user

[Paul Russell]

Hi Brian,

Thanks for this. I've made some comments below...

Post by Brian Topping

[...] I come from a websphere
process choreographer background, which works using database-based
persistence to store state, and JMS + Message Driven Beans to provide
activity scheduling and dispatch - I was kind of /assuming/ a similar
architecture for implementing BPEL.

Wrapping Werkflow with this functionality should not be difficult.
Repository interfaces exist inside the system, so it's really just a matter
of extending your persistence mechanism to include them on the back end.
Adding the MDB transport on the front end should be a couple dozen lines of
code.

I'm not sure we're talking about quite the same thing here. Although
IBM's framework /can/ use JMS as a framework for controlling the
workflow container (including starting processes etc), it also uses it
internally to schedule the transitions between activities for
macro-flows. This allows one process to run across many nodes (although
usually, the EJB container's affinity will prefer each activity to run
on the same node as the last). If the node fails for any reason, the
process will forward recover on another node, since the message that
triggered the last activity will be 'rolled back' onto the queue.

Post by Brian Topping

I guess what I'm trying to assess is how far off being
production ready
for werkflow is, both in a generic sense and in a BPEL sense. So for
* Support running in a multi-node environment
* At least with High Availability.
* Preferably with load balancing and high availability.

This probably depends more on your container than Werkflow. If you had a set
of routines that you wanted to do this, how would you do it? Werkflow just
provides the entry points for you to call. It's very independent of
container, so you don't have to worry about all the headaches of the engine
playing with the container state.

You're absolutely correct that Werkflow shouldn't be directly involved
in this. However, as it stands, as far as I know, a process can't be
failed over half way through. This is fine for processes that take a
few seconds to execute and return a result, but if a process takes
weeks or even months, this is a bit of a killer for the application I
have in mind.

Post by Brian Topping

* Support graphical debugging through development tooling.

This is one that I am particularly interested in myself. It's not there yet.
I'm looking at generating my source files from XMI, and I could imagine that
there would be a way to have a fat client that could graphically trace the
UML that was in the XMI. I have some ideas here, but I don't really
understand what ought to happen because I've never seen a commercial workflow
engine in operation.

Yeah, that's absolutely the kind of thing I'm thinking about. I'm not
sure XMI will cover it directly, since BPEL doesn't map as directly
onto UML as some of the other 'flow' based languages (IBM's FDML, etc).
One thing that occurred to me is that if we could implement some kind
of traceability between the 'core' workflow language and the
personality, it might be possible to derive the current node in the
personality workflow based on that traceability.

Post by Brian Topping

* Support development tooling ;)

Would generation from UML work?

It certainly could (should), although there is a bit of a mismatch
here. UML Activity Diagrams (with a few extensions for supporting
compensation etc) map pretty well onto the 'flow' markup within BPEL.
The more procedural constructs like 'sequence' 'choice' etc don't map
so well onto UML. You could conceivably use sequence or collaboration
diagrams, but the match would be much more approximate than for <flow>.

Post by Brian Topping

Do you guys have any thoughts as to whether these are things you were
expecting to support as part of werkflow? If so, do you have
a road map
to get there? I'm quite willing to help if you can point me in the
right direction!

I can't speak for the rest of the team, but I am working on these as a part
of my project. It's just a matter of time. But that still doesn't answer
your question. Do you have the resources to help with this or are you
needing something that is more complete from the outset?

At the moment, I'm in the investigation stage of a project (i.e. budget
has not yet been allocated). Should a decision be taken to proceed,
then funds would be available to directly contribute, assuming you guys
are open to this.

Post by Brian Topping
I'm personally not that impressed by the other solutions out there, and feel
like I'd rather invest in Werkflow than try to fix what's broken in the
backend on the other tools.

Agreed. I'm trying to work out what changes would have to be make to
Werkflow to support what I'm talking about here, to work out whether
it's viable to modify Werkflow, or whether the concepts are just too
different. Right now, my instinct says Werkflow provides a good
framework to build on, but that there might be quite a lot of work to
do to make it do all the things I'm looking for.

Post by Brian Topping
If/when I attack this, it will probably be as I described above. It would be
great to have you help with rough text or graphic descriptions of what you
are looking for. Can you provide some ideas?

Certainly. Happy to help in any way I can. I'm about to send a separate
e-mail outlining one of the things I've been thinking about in more
detail. I'd welcome feedback, comments and common sense!!

Paul

[Mark H. Wilkinson]

Post by Paul Russell

Post by Brian Topping

[...] I come from a websphere
process choreographer background, which works using database-based
persistence to store state, and JMS + Message Driven Beans to provide
activity scheduling and dispatch - I was kind of /assuming/ a similar
architecture for implementing BPEL.

Adding the MDB transport on the front end should be a couple dozen
lines of code.

I'm not sure we're talking about quite the same thing here. Although
IBM's framework /can/ use JMS as a framework for controlling the
workflow container (including starting processes etc), it also uses it
internally to schedule the transitions between activities for
macro-flows. This allows one process to run across many nodes (although
usually, the EJB container's affinity will prefer each activity to run
on the same node as the last). If the node fails for any reason, the
process will forward recover on another node, since the message that
triggered the last activity will be 'rolled back' onto the queue.

I see what you're getting at, and I'm not sure how simple it would be to
hook that kind of functionality into werkflow. At the moment the core of
the engine uses Doug Lea's concurrent library to provide a simple thread
pooling system for scheduling activities. Currently this means that it
wouldn't be possible to run the werkflow core within an EJB container,
for example. That's not been a great problem up-to-date though - EJB
containers don't really provide much in the way of support for building
multithreaded applications.

The J2EE 1.4 spec adds a timer service to the EJB container, so with
that it should be possible to interface werkflow to an EJB container.
I'm thinking that this would be a case of splitting off the thread
scheduling part of werkflow into a pluggable interface. By doing that
we'd probably be able to schedule activities using JMS messages, the EJB
timer service, a simple thread pool, or whatever.

I'd been thinking about resilience and fail-over a bit myself: the
project I'm working on at the moment would need these kinds of features
in the future. My starting point would really be getting werkflow to
persist process cases into a relational database reliably (i.e. making
activity execution a transaction against the database), on the basis
that building resilient databases is a solvable problem. Once that's
done you should actually be able to cluster werkflow - you'd be using
database locks to make sure that only one instance of werkflow could run
a given activity at a given time.

One way of looking at it is that IBM's product is using JMS messages to
hold process case state, while werkflow is more likely to use a generic
relational database.

Post by Paul Russell
You're absolutely correct that Werkflow shouldn't be directly involved
in this. However, as it stands, as far as I know, a process can't be
failed over half way through. This is fine for processes that take a
few seconds to execute and return a result, but if a process takes
weeks or even months, this is a bit of a killer for the application I
have in mind.

Yes, at the moment werkflow can't do this. The persistence interfaces
need completing so that the state of the werkflow engine can be held on
disc rather than in memory. I definitely want to get this working
though.

-Mark.

[Paul Russell]

Hi Mark,

Post by Mark H. Wilkinson
One way of looking at it is that IBM's product is using JMS messages to
hold process case state, while werkflow is more likely to use a generic
relational database.

Interestingly IBM's product doesn't work quite like this. It /does/ use
a database to persist 'case' (they call them process instance) state.
JMS only gets involved for the 'control' links, i.e. scheduling what to
run next (or in parallel). As I say, more on this later when I get time
to finish my other e-mail ;)

Paul

[Brian Topping]

Hi Paul,

OIC... like I said, I have noooo experience with commercial workflow stuff.
heh!

That's a pretty slick idea for clustering the internals of the workflow
engine. I think it would probably take a day of fiddling, but it would be
nice to see if the calls for that can be done through an interface to make
the work (un)pluggable. It's nice to be without all that container noise
when you are keeping things lightweight.

I started a UML transform a few hours ago, but I'm not going to have it
finished tonight. Things look good with it though, I'll keep you up to date.

As for the debugging engine, I'd honestly like to look at building a Swing
client that used the novosoft UML library (or even just iterating a SVG
exported for the purpose) for graphical indication of where the process is.

What I am talking about here might be too low tech for what you are doing,
but...

-b

-----Original Message-----
Sent: Tuesday, February 24, 2004 6:58 PM
Subject: Re: [werkflow-user] BPEL support
Hi Brian,
Thanks for this. I've made some comments below...

Post by Brian Topping

[...] I come from a websphere
process choreographer background, which works using database-based
persistence to store state, and JMS + Message Driven Beans

to provide

Post by Brian Topping

activity scheduling and dispatch - I was kind of

/assuming/ a similar

Post by Brian Topping

architecture for implementing BPEL.

Wrapping Werkflow with this functionality should not be difficult.
Repository interfaces exist inside the system, so it's

really just a

Post by Brian Topping
matter
of extending your persistence mechanism to include them on the back
end.
Adding the MDB transport on the front end should be a couple dozen
lines of
code.

I'm not sure we're talking about quite the same thing here. Although
IBM's framework /can/ use JMS as a framework for controlling the
workflow container (including starting processes etc), it
also uses it
internally to schedule the transitions between activities for
macro-flows. This allows one process to run across many nodes
(although
usually, the EJB container's affinity will prefer each
activity to run
on the same node as the last). If the node fails for any reason, the
process will forward recover on another node, since the message that
triggered the last activity will be 'rolled back' onto the queue.

Post by Brian Topping

I guess what I'm trying to assess is how far off being
production ready
for werkflow is, both in a generic sense and in a BPEL

sense. So for

Post by Brian Topping

example, for me to consider this production ready, I'd
* Support running in a multi-node environment
* At least with High Availability.
* Preferably with load balancing and high availability.

This probably depends more on your container than Werkflow. If you
had a set
of routines that you wanted to do this, how would you do

it? Werkflow

Post by Brian Topping
just
provides the entry points for you to call. It's very independent of
container, so you don't have to worry about all the

headaches of the

Post by Brian Topping
engine
playing with the container state.

You're absolutely correct that Werkflow shouldn't be directly
involved
in this. However, as it stands, as far as I know, a process can't be
failed over half way through. This is fine for processes that take a
few seconds to execute and return a result, but if a process takes
weeks or even months, this is a bit of a killer for the application I
have in mind.

Post by Brian Topping

* Support graphical debugging through development tooling.

This is one that I am particularly interested in myself. It's not
there yet.
I'm looking at generating my source files from XMI, and I could
imagine that
there would be a way to have a fat client that could

graphically trace

Post by Brian Topping
the
UML that was in the XMI. I have some ideas here, but I don't really
understand what ought to happen because I've never seen a

commercial

Post by Brian Topping
workflow
engine in operation.

Yeah, that's absolutely the kind of thing I'm thinking about. I'm not
sure XMI will cover it directly, since BPEL doesn't map as directly
onto UML as some of the other 'flow' based languages (IBM's
FDML, etc).
One thing that occurred to me is that if we could implement some kind
of traceability between the 'core' workflow language and the
personality, it might be possible to derive the current node in the
personality workflow based on that traceability.

Post by Brian Topping

* Support development tooling ;)

Would generation from UML work?

It certainly could (should), although there is a bit of a mismatch
here. UML Activity Diagrams (with a few extensions for supporting
compensation etc) map pretty well onto the 'flow' markup within BPEL.
The more procedural constructs like 'sequence' 'choice' etc don't map
so well onto UML. You could conceivably use sequence or collaboration
diagrams, but the match would be much more approximate than
for <flow>.

Post by Brian Topping

Do you guys have any thoughts as to whether these are

things you were

Post by Brian Topping

expecting to support as part of werkflow? If so, do you have
a road map
to get there? I'm quite willing to help if you can point me in the
right direction!

I can't speak for the rest of the team, but I am working on

these as a

Post by Brian Topping
part
of my project. It's just a matter of time. But that still doesn't
answer
your question. Do you have the resources to help with this

or are you

Post by Brian Topping
needing something that is more complete from the outset?

At the moment, I'm in the investigation stage of a project
(i.e. budget
has not yet been allocated). Should a decision be taken to proceed,
then funds would be available to directly contribute,
assuming you guys
are open to this.

Post by Brian Topping
I'm personally not that impressed by the other solutions out there,
and feel
like I'd rather invest in Werkflow than try to fix what's

broken in the

Post by Brian Topping
backend on the other tools.

Agreed. I'm trying to work out what changes would have to be make to
Werkflow to support what I'm talking about here, to work out whether
it's viable to modify Werkflow, or whether the concepts are just too
different. Right now, my instinct says Werkflow provides a good
framework to build on, but that there might be quite a lot of work to
do to make it do all the things I'm looking for.

Post by Brian Topping
If/when I attack this, it will probably be as I described

above. It

Post by Brian Topping
would be
great to have you help with rough text or graphic

descriptions of what

Post by Brian Topping
you
are looking for. Can you provide some ideas?

Certainly. Happy to help in any way I can. I'm about to send
a separate
e-mail outlining one of the things I've been thinking about in more
detail. I'd welcome feedback, comments and common sense!!
Paul
--
Paul Russell
_______________________________________________
werkflow-user mailing list
http://lists.codehaus.org/mailman/listinfo/werkflow-user

[Paul Russell]

Hiya,

I got half-way through another e-mail about the scheduling stuff last
night outlining what I thought might be a useful approach.
Unfortunately it hit 1am over here, and I thought I should probably get
to bed! I'll try and finish it sometime today (UK time). Totally agree
that we wouldn't want to put all the heavy-weight scheduling stuff in
for everyone -- my gut instinct says we should turn it into a service
like the messaging and persistence ones. More on this later!

Paul

Post by Brian Topping
's a pretty slick idea for clustering the internals of the workflow
engine. I think it would probably take a day of fiddling, but it would be
nice to see if the calls for that can be done through an interface to make
the work (un)pluggable. It's nice to be without all that container noise
when you are keeping things lightweight.