SOA and "White box" code reuse...

[Winfield Hill]

http://www-128.ibm.com/developerworks/webservices/library/ws-reuse-soa.html

The service interface is the essence of the integration design. Combined
with the use of standards, interfaces are the essential ingredient for
creating a loose coupling where service clients and service providers
can communicate regardless of programming language and platform. Services
are to be independent, in that clients need not understand the inner
workings of a service component; essentially the service operates as a
"black box." "White box" reuse, or cut and paste, where source code is
modified in order to use in another context, while useful, is not
typically as beneficial as "black box reuse." "Systematic reuse programs
encourage reusing software without change because of the superior benefit
they receive from black-box reuse throughout the life cycle."

 

[The Real Andy]

http://www-128.ibm.com/developerworks/webservices/library/ws-reuse-soa.html

The service interface is the essence of the integration design. Combined
with the use of standards, interfaces are the essential ingredient for
creating a loose coupling where service clients and service providers
can communicate regardless of programming language and platform. Services
are to be independent, in that clients need not understand the inner
workings of a service component; essentially the service operates as a
"black box." "White box" reuse, or cut and paste, where source code is
modified in order to use in another context, while useful, is not
typically as beneficial as "black box reuse." "Systematic reuse programs
encourage reusing software without change because of the superior benefit
they receive from black-box reuse throughout the life cycle."

And?? I have been programming SOA since the mide 90's on an enterprise
level system. To date, i haver never worked on a system that is as
flexible/scalable as that one.

Microsoft is beggining to push SOA with there new OS. Indigo (Windows
Communication Foundation) ships with Winows Vista and libraries are
available for XP, 2000 and I think also NT. I have had a play with
Indigo on Vista and it is very impressive. I queried one of the
Product Managers from MS on real time message processing capabilites.
He pointed me to some papers and informed me that one of the test
cases is on a high pressure oil pipeline, where they use indigo to
commuincate with sensors in real time. A few of the examples also deal
with real time process control.

 

[Winfield Hill]

The Real Andy wrote...

And?? I have been programming SOA since the mide 90's on an enterprise
level system. To date, i haver never worked on a system that is as
flexible/scalable as that one.

I guess the question many of us would have, is how does one learn to
program using Service-Oriented Architecture techniques? It sounds
good, but it there a formal methodology? Are there useful easily-
understood rules to creating and using SOA, say in an embedded system?
It appears that most of the "software reuse" books that IBM references
are serious head-nodding-off eyelid-droopers. :>)

And then there's SOAD, Service-Oriented Analysis and Design, and
Service-oriented modeling and architecture, and don't forget SOAP,
Simple Object Access Protocol. See http://www.w3.org/TR/soap/

There's also NASSL, Network Accessible Service Specification Language
and WSDL, Web Services Description Language and WIDL, the Web Interface
Definition Language. OK, fine. We'll just roll this all up into one
big XML ball of wax. <sigh>

 

[Frithiof Andreas Jensen]

http://www-128.ibm.com/developerworks/webservices/library/ws-reuse-soa.html

The service interface is the essence of the integration design. Combined
with the use of standards, interfaces are the essential ingredient for
creating a loose coupling where service clients and service providers
can communicate regardless of programming language and platform. Services
are to be independent, in that clients need not understand the inner
workings of a service component; essentially the service operates as a
"black box." "White box" reuse, or cut and paste, where source code is
modified in order to use in another context, while useful, is not
typically as beneficial as "black box reuse." "Systematic reuse programs
encourage reusing software without change because of the superior benefit
they receive from black-box reuse throughout the life cycle."

Fine words indeed - these things always emerge on regular cycles before
collapsing into complexity and chaos. Managers like it because they alway
hope that they can get some monkey with a smart tool to produce quality work
for peanuts. And the "Hello Word" application does indeed look cool.

The problem is that once "the architects" have finished adding
layer-upon-layer of framework, interfaces, fascades and protocols (and
helper's helper services), the requirements of "The Job" will be so deeply
buried under those of "The Tool" that very little Work is actually achieved
and everybody would have been more happy if processes just send each other
email ;-).

Take CORBA with IPv6 f.ex. - once one has worked around the combined
brokeness of all the different components, ORB's, programming languages and
platforms in the project, the only excuse left for using CORBA is that a)
the customer likes it enough to pay in excess, b) looks good on the CV when
leaving the project (Like that management system with 60 MB of tools and 6
MB of Job, on a hardware-limited platform - I have had the "honour" to have
been involved with).

RMI, regardless of the wrapping it comes in, is just Evil.

 

[Winfield Hill]

Frithiof Andreas Jensen wrote...

Fine words indeed - these things always emerge on regular cycles
before collapsing into complexity and chaos. Managers like it
because they alway hope that they can get some monkey with a smart
tool to produce quality work for peanuts. And the "Hello Word"
application does indeed look cool.

The problem is that once "the architects" have finished adding
layer-upon-layer of framework, interfaces, fascades and protocols
(and helper's helper services), the requirements of "The Job" will
be so deeply buried under those of "The Tool" that very little Work
is actually achieved and everybody would have been more happy if
processes just send each other email ;-). [ snip ]

Linus Torvalds, on specifications, http://kerneltrap.org/node/5725
"specs have an inevitable tendency to try to introduce abstractions
levels and wording and documentation policies that make sense for a
written spec. Trying to implement actual code off the spec leads to
the code looking and working like CRAP."

 

[The Real Andy]

Fine words indeed - these things always emerge on regular cycles before
collapsing into complexity and chaos. Managers like it because they alway
hope that they can get some monkey with a smart tool to produce quality work
for peanuts. And the "Hello Word" application does indeed look cool.

The problem is that once "the architects" have finished adding
layer-upon-layer of framework, interfaces, fascades and protocols (and
helper's helper services), the requirements of "The Job" will be so deeply
buried under those of "The Tool" that very little Work is actually achieved
and everybody would have been more happy if processes just send each other
email ;-).

That can be said for any programming pattern. I think the problems
stems from "architects" not putting enough though into the problem and
by failing to understand the business logic. I have been fortunate to
work with a good architech who designed a system from the ground up,
that is today still ahead of its time and so simple to program that a
uni grad can develop good apps within a few months.

I think there is another problem in that sometimes SOA's are used when
quite clearly there are much better and simpler options. With SOA
being the current buzzword, the managers and architects love it, and
most use it incorrectly

Take CORBA with IPv6 f.ex. - once one has worked around the combined
brokeness of all the different components, ORB's, programming languages and
platforms in the project, the only excuse left for using CORBA is that a)
the customer likes it enough to pay in excess, b) looks good on the CV when
leaving the project (Like that management system with 60 MB of tools and 6
MB of Job, on a hardware-limited platform - I have had the "honour" to have
been involved with).

RMI, regardless of the wrapping it comes in, is just Evil.

Being a c++ programmer i am not familiar with RMI, but from what i can
gather is that it is similar to MS's RPC, please correct me if I am
wrong.

 

[Frithiof Andreas Jensen]

On Mon, 3 Oct 2005 12:52:17 +0200, "Frithiof Andreas Jensen"


Being a c++ programmer i am not familiar with RMI, but from what i can
gather is that it is similar to MS's RPC, please correct me if I am
wrong.

Remote Method Invocation - The New & Improved O.O. name for ..... TaDaaa
..... old and crufty Remote Procedure Calls which, as far as I recall, SUN
re-invented and Microsoft then saw as a must-also-have feature back when it
was Hot; i.e. when it was merely "brochure-ware" only.

Before the proliferations of application.... errr. worms *using* that
particular interface.

 

[The Real Andy]

The Real Andy wrote...

I guess the question many of us would have, is how does one learn to
program using Service-Oriented Architecture techniques? It sounds
good, but it there a formal methodology? Are there useful easily-
understood rules to creating and using SOA, say in an embedded system?
It appears that most of the "software reuse" books that IBM references
are serious head-nodding-off eyelid-droopers. :>)

Pfft, IBM.. Prohibitavly expensive with an outcome that always ends up
in tears

TO be honest though, the books are good. Like any acedemic book it
will be hard to read from cover to cover. There is a plethora of good
books on architecture patterns. Whilst I know I will get ripped for
talking up MS, www.microsoft.com/patterns has a few really good papers
(books) on architecture. The papers may discuss MS product, but the
underlying concepts remain the same across any platform. Off topic,
but we can blame billy for his success, but the fact is that MS is run
by lawers and accountants these days, but in the heart there is some
really talented programmers. Those papers are worth a look.

And then there's SOAD, Service-Oriented Analysis and Design, and
Service-oriented modeling and architecture, and don't forget SOAP,
Simple Object Access Protocol. See http://www.w3.org/TR/soap/

I think you will find SOAP is becoming increasingly popular due to its
simplicity.

There's also NASSL, Network Accessible Service Specification Language
and WSDL, Web Services Description Language and WIDL, the Web Interface
Definition Language. OK, fine. We'll just roll this all up into one
big XML ball of wax. <sigh>

All this stuff normally gets layered over soap The bonus with XML
is that you dont run into versioning problems. Older services get the
feilds they need, Newer services can add extra feilds into messages
that are still backward compatible. The other bonus with these
standards is that they interoperate with multiple platforms.

I hate to talk MS up again (I write windows software, so its all I
know these days ) but may i also suggest David Pallmann's book
Programming Indigo. I think Indigo will be a model for a lot of OS's
soon. It is very flexible yet makes it ever so simple to implement
SOA. As a bonus it also encourages good programming.

At the end of the day, SOA for architecture is like OO for software.
Its very powerful, but can easily be misused and misunderstood.


Please excuse any incoherence in my posts, i just started on nicotine
patches today and its having a bizzare impact on my body.

 

[John Woodgate]

I read in sci.electronics.design that Winfield Hill

Trying to implement actual code off the spec leads to
the code looking and working like CRAP."

Code Really Anti-Productive?

 

[Paul Hovnanian P.E.]

http://www-128.ibm.com/developerworks/webservices/library/ws-reuse-soa.html

The service interface is the essence of the integration design. Combined
with the use of standards, interfaces are the essential ingredient for
creating a loose coupling where service clients and service providers
can communicate regardless of programming language and platform. Services
are to be independent, in that clients need not understand the inner
workings of a service component; essentially the service operates as a
"black box." "White box" reuse, or cut and paste, where source code is
modified in order to use in another context, while useful, is not
typically as beneficial as "black box reuse." "Systematic reuse programs
encourage reusing software without change because of the superior benefit
they receive from black-box reuse throughout the life cycle."

Very nice, but code re-use is a pretty poor argument for using SOA.
Heck, we've had code re-use with "standard interfaces" (we call them
APIs) since the early days of static or dynamic library linking.

There are good reasons to use SOA, but re-use can be achieved with far
less cost and pain.

 

[The Real Andy]

Very nice, but code re-use is a pretty poor argument for using SOA.
Heck, we've had code re-use with "standard interfaces" (we call them
APIs) since the early days of static or dynamic library linking.

API, COM Service, same shit. The difference being how you interface.
With an API or COM, there is a strictly defined interface.

With Services the interface is loosey coupled and autonomous. Changing
a service has no affect on how another service runs. Likewise, a
services interface can change along with the message and this will not
affect another service or backward compatibility.

There are good reasons to use SOA, but re-use can be achieved with far
less cost and pain.

Being the buzzword it is, i think people need to step back and
understand, or at least read up on the topic. I think unless you are
developing enterprise level systems then its probably a waste of time
using a SOA.

 

[The Real Andy]

Remote Method Invocation - The New & Improved O.O. name for ..... TaDaaa
.... old and crufty Remote Procedure Calls which, as far as I recall, SUN
re-invented and Microsoft then saw as a must-also-have feature back when it
was Hot; i.e. when it was merely "brochure-ware" only.

Before the proliferations of application.... errr. worms *using* that
particular interface.

RPC has been tightened up these days. That has been a problem for me
because all the stuff I do now is based on RPC. Whilst there is a way
to switch of the security it is a poor solution. At the end of the
day, i thought RPC was a pretty neat solution for distributed
components models, but perhaps that the sicko c++ side of me coming
out !

I think if I was to do it again i would go for WSE in SOAP. Only
problem here is bandwidth. Back when I first started distributed
programming, our telco could only provide 1200baud comms reliable
enough for commercial use. Now, they can provide a min of 14k to all
rural areas over a copper pair using IPWan or IPDirect. If you are
lucky you will get upto 56k.

The bandwith problem made us think up a modular yet minimalist
approach. Feild upgrades over the wire were essential. So in the end a
SOA won over. All base class libraries were written from the ground up
to escape from the code/resource heavy MFC and STL and all protocols
where designed to be minimal.

 

[Frithiof Andreas Jensen]

API, COM Service, same shit. The difference being how you interface.
With an API or COM, there is a strictly defined interface.

With Services the interface is loosey coupled and autonomous. Changing
a service has no affect on how another service runs. Likewise, a
services interface can change along with the message and this will not
affect another service or backward compatibility.



Being the buzzword it is, i think people need to step back and
understand, or at least read up on the topic. I think unless you are
developing enterprise level systems then its probably a waste of time
using a SOA.

I must say that the SOA concept itself is very effective, it is often used
in embedded system under the disguise of Mailbox interproccess communication
where it solves a lot of hard problems.

IMO it's the bloated SOA *Framework's* that effectively kill any potential
advantages of SOA for more distributed tasks!

The general problem with the available frameworks is that architects try to
imagine all possible uses and scenarios and accomodate them all in one
package and then in Real Life you find that the actual needs are different
and the messaging you need to really do incredibly useful work are actually
a very simple data exchange with very simple formats.

But now you have commited to a framework and you are lumbered with the task
of satisfying more Framework needs than Application needs just to get even
the simplest application to run!!

For Example: CORBA forces TCP/IP and TCP/IP is absolute shite with ad-hoc
networks were the routing can change often (mostly thanks to optimisations
in the Linux kernel that are really hard to hack out - and - yet to be
discovered - *other stuff* can be sure to rely on the behaviour thus
enshrined in the holy Kernel tables ;-).

In my experience frameworks need to be Grown more than Designed: it is just
a total waste to try and invent a complete solution for applications that
one has not even run yet.

The job of the architect, IMO, is to provide the minimum possible "framework
kernel" that can still evolve painlessly as experience is gained with real
applications of it. Not to try an solve every imagined problem.


We successfully used a framework called "opensis" for this project:
http://www.b2ncw.org .

However, and this is *my* opinion, If *I* were ever to do something like
this again, I would definitely drop all that multi-tiered CORBA framework
crap and the mostly imagined "Interoperability" requirement with every
platform ever concieved and go for an extendable mailbox-like
Consumer/Subscriber interface based on pure textual(!) messaging, layered on
top of an effective multicast transport protocol like SPREAD and on ONE
platform too.



PS:

I very much *LIKE* textual formats - it *forces* people to keep their
interfaces opaque; I can debug it from a terminal, which often is what I
happen to have; It is easier to understand what goes on in an interface when
one can read it; It is easy to add new commands/fields because the parser of
course will be designed to ignore what it does not understand; and Testing
is easier too because you need a very minimal program to do it with - f.ex.
an "expect" script.

Most of the time there is no performance reasons whatsoever to use Binary
formats (if one worry about the link capacity text will, unlike binary,
compress well).

 

[[email protected]]

awk '{ print $3, $5, $7 }' <input.txt | sort | uniq -c | sort -n

OK, this stays on one platform, but you could include an ssh to another
box in there if you wanted to.

The bonus with XML is that you dont run into versioning problems. Older
services get the feilds they need, Newer services can add extra feilds
into messages that are still backward compatible.

struct message_header
{
uint8 type;
uint8 length;
};

while(!eof)
{
read type;
read length;
read "length" bytes into buffer
switch(type)
{
case 1:
deal_with_case_1;
break;
case 2:
deal_with_case_2;
break;
case 3:
deal_with_case_3;
break;
default:
sigh_deeply_and_ignore_message;
}
}

But yeah, fread() and fseek() are a lot more complicated than an XML
parser.

I think Indigo will be a model for a lot of OS's soon.

Totally new operating systems are designed every single day? Wow!

As a bonus it also encourages good programming.

ld: undefined reference "it", possible matches in:
libstructuredprogramming.so
libpascal.so
libobjectorientedprogramming.so
libvisualbasic.so
libjava.so

Of course, the key indicator is the face of the person who is developing
Indigo (or SOA, or XML, or whatever.) The law is clear: "There is a
beard - there is a success. There is no beard - you are guilty."
http://khason.biz/blog/2004/12/why-microsoft-can-blow-off-with-c.html

Matt Roberds

 

[The Real Andy]

awk '{ print $3, $5, $7 }' <input.txt | sort | uniq -c | sort -n

OK, this stays on one platform, but you could include an ssh to another
box in there if you wanted to.


struct message_header
{
uint8 type;
uint8 length;
};

while(!eof)
{
read type;
read length;
read "length" bytes into buffer
switch(type)
{
case 1:
deal_with_case_1;
break;
case 2:
deal_with_case_2;
break;
case 3:
deal_with_case_3;
break;
default:
sigh_deeply_and_ignore_message;
}
}

Take a long time to think that up?

But yeah, fread() and fseek() are a lot more complicated than an XML
parser.

Standards make communicating easy. I was referring to standards. As a
bonus, most OS's have standards for common libraries.

A typical academic programmer will say 'roll your own'. A typical
commercially successfull programmer says 'why roll your own when some
has already done it for you?'. A bit like writing your own database.

Totally new operating systems are designed every single day? Wow!


ld: undefined reference "it", possible matches in:
libstructuredprogramming.so
libpascal.so
libobjectorientedprogramming.so
libvisualbasic.so
libjava.so

Of course, the key indicator is the face of the person who is developing
Indigo (or SOA, or XML, or whatever.) The law is clear: "There is a
beard - there is a success. There is no beard - you are guilty."
http://khason.biz/blog/2004/12/why-microsoft-can-blow-off-with-c.html

That link provided a terrific argument.


You sound like a dedicated linux programmer.

 

[Paul Hovnanian P.E.]

I must say that the SOA concept itself is very effective, it is often used
in embedded system under the disguise of Mailbox interproccess communication
where it solves a lot of hard problems.

IMO it's the bloated SOA *Framework's* that effectively kill any potential
advantages of SOA for more distributed tasks!

The general problem with the available frameworks is that architects try to
imagine all possible uses and scenarios and accomodate them all in one
package and then in Real Life you find that the actual needs are different
and the messaging you need to really do incredibly useful work are actually
a very simple data exchange with very simple formats.

But now you have commited to a framework and you are lumbered with the task
of satisfying more Framework needs than Application needs just to get even
the simplest application to run!!

For Example: CORBA forces TCP/IP and TCP/IP is absolute shite with ad-hoc
networks were the routing can change often (mostly thanks to optimisations
in the Linux kernel that are really hard to hack out - and - yet to be
discovered - *other stuff* can be sure to rely on the behaviour thus
enshrined in the holy Kernel tables ;-).

In my experience frameworks need to be Grown more than Designed: it is just
a total waste to try and invent a complete solution for applications that
one has not even run yet.

The job of the architect, IMO, is to provide the minimum possible "framework
kernel" that can still evolve painlessly as experience is gained with real
applications of it. Not to try an solve every imagined problem.

Yeahbut... The frameworks were developed to try and cover a wide range
of general cases. Then, the designers and architects sat down and
implemented them.

Management is being sold SOA for 'code reuse' (in the OP's article), so
if you start talking about growing frameworks, management starts getting
nervous thinking that you are about to re-code something they thought
was already complete.

The CORBA and TCP/IP example is a good one. CORBA was written to include
interprocess communication (IPC) between different hosts. If you've got
all your processes running in one box (or on one chip), why carry around
all the networking overhead?

Ideally, a framework would be developed that would allow local processes
to bypass parts of the protocol stack that are not needed. Example:
socket-based IPC can use either an INET or a UNIX (pipe) protocol. A
well build implementation of a framework should allow the user (those
coding to that framework) the flexibility of selecting the appropriate
protocol. Many do not.

We successfully used a framework called "opensis" for this project:
http://www.b2ncw.org .

However, and this is *my* opinion, If *I* were ever to do something like
this again, I would definitely drop all that multi-tiered CORBA framework
crap and the mostly imagined "Interoperability" requirement with every
platform ever concieved and go for an extendable mailbox-like
Consumer/Subscriber interface based on pure textual(!) messaging, layered on
top of an effective multicast transport protocol like SPREAD and on ONE
platform too.

PS:

I very much *LIKE* textual formats - it *forces* people to keep their
interfaces opaque; I can debug it from a terminal, which often is what I
happen to have; It is easier to understand what goes on in an interface when
one can read it; It is easy to add new commands/fields because the parser of
course will be designed to ignore what it does not understand; and Testing
is easier too because you need a very minimal program to do it with - f.ex.
an "expect" script.

Most of the time there is no performance reasons whatsoever to use Binary
formats (if one worry about the link capacity text will, unlike binary,
compress well).

I like them as well. Using telnet to the appropriate port is a great
debugging tool. Generic XML debugging tools are available all over the
place (for free, too). But if you're in the business of selling
development environments (many $$ per seat), they are evil. You know
what they say about one man's evil.

 

[Clifford Heath]

Ideally, a framework would be developed that would allow local processes
to bypass parts of the protocol stack that are not needed. Example:
socket-based IPC can use either an INET or a UNIX (pipe) protocol. A
well build implementation of a framework should allow the user (those
coding to that framework) the flexibility of selecting the appropriate
protocol. Many do not.

That's exactly what WCF (aka Indigo) does. It seems to me to
be a rare example of an exceptionally fine design by MS...
and I've designed and built more than one communication
framework during my career .

 

[John Larkin]

http://www-128.ibm.com/developerworks/webservices/library/ws-reuse-soa.html

The service interface is the essence of the integration design. Combined
with the use of standards, interfaces are the essential ingredient for
creating a loose coupling where service clients and service providers
can communicate regardless of programming language and platform. Services
are to be independent, in that clients need not understand the inner
workings of a service component; essentially the service operates as a
"black box." "White box" reuse, or cut and paste, where source code is
modified in order to use in another context, while useful, is not
typically as beneficial as "black box reuse." "Systematic reuse programs
encourage reusing software without change because of the superior benefit
they receive from black-box reuse throughout the life cycle."

"The reuser (service client) does not really even know what code they
are reusing, and don’t really care."

"As an example, consider common capability needs such as user access
revalidation for applications within an enterprise."

With concepts and prose as sloppy as this, it's no wonder we live in
the Dark Ages of software.

John

 

[[email protected]]

Take a long time to think that up?
Weeks.

Standards make communicating easy.

In general, I agree. But there is a balance that has to be struck
between creating a standard that will cover every single case vs. one
that will actually be useful. OSI vs TCP/IP might be a good example.

A typical academic programmer will say 'roll your own'. A typical
commercially successfull programmer says 'why roll your own when some
has already done it for you?'.

This can work well when the person that did it for you knew what they
were doing. Much of the time, this is the case. Sometimes it's not and
it can make you crazy.

A bit like writing your own database.

At one time, I was engaged in writing an AR/AP program from scratch in
Java. (The project wasn't advertised that way, but that's what it turned
into.) My comments to management that this was a rather silly thing to
do went unheeded. That company went bankrupt.

You sound like a dedicated linux programmer.

I have been paid to program on VMS, Windows, VxWorks, an embedded Z80
derivative that came with a C-ish compiler, and several Unices, including
AIX, SunOS, Solaris, HP/UX, CX/UX. Oh yeah, and also Linux.

Matt Roberds