Re: Topology Service losing events when there is port congestion

So I just have a follow up question about topology service congestion.  If we have one node that has a
TIPC_FLOW_CONTROL_WIN set to 512 and the other node with a TIPC_FLOW_CONTROL_WIN set to 3000, we'll end
up in a deadlock situation when the first node sends 1024 (twice the TIPC_FLOW_CONTROL_WIN), but the
receiving side only thinks it has to ack every 3000 messages.  Is there no timer element involved in this? 
What if you have a system that sent only 400 messages in an hour?  No ack will be sent in this case?

What kind of timeout message handling (waiting for an ack), if any, is incorporated into to TIPC).

Thanks,
Felix

-----Original Message-----
From: Jon Maloy [mailto:jon.maloy <at> ericsson.com]
Sent: Thursday, June 02, 2011 6:21 PM
To: Felix Nayman; Stephens, Allan; tipc-discussion <at> lists.sourceforge.net
Subject: RE: [tipc-discussion] Topology Service losing events when there is port congestion

Hi Felix,
I didn't remember that mail exhange, but it seems like the suggestion was very similar to my proposal from
last Friday's mail.

Basically, as a quick fix, you can do either

static inline int tipc_port_congested(struct port *p_ptr) {
        return((p_ptr->sent - p_ptr->acked) >= ((TIPC_FLOW_CONTROL_WIN * 2) +
p_ptr->topology_server_set_adjustment_constant /* 0 for all other ports */ )); }

Or maybe better:

static inline int tipc_port_congested(struct port *p_ptr) {
        return((p_ptr->sent - p_ptr->acked) >= p_ptr->flow_control_limit);
}

The new 'flow_control_limit' is initialized to ((TIPC_FLOW_CONTROL_WIN * 2) in all ports, while
the topology server overwrites it to something much bigger in its own server ports.
No setsockopt() or anything else involved here, and I can hardly imagine any simpler solution.

Of course, in the longer run we should find some way to convey the overload limit of the peer port and use that,
but that requires a protocol update, with all the backwards compatibility considerations etc that such a
change entails.

Why don't you just try it out and report the result?

Regards
///jon

Jon Maloy M.Sc. EE
Researcher
Ericsson Canada
Broadband and Systems Research
8400 Decarie
H4P 2N2, Montreal, Quebec, Canada
Phone + 1 514 345-7900 x42056
Mobile + 1 514 591-5578
jon.maloy <at> ericsson.com
www.ericsson.com

-----Original Message-----
From: Felix Nayman [mailto:Felix.Nayman <at> infinite.com]
Sent: June-02-11 15:26
To: Felix Nayman; Jon Maloy; Stephens, Allan; tipc-discussion <at> lists.sourceforge.net
Subject: RE: [tipc-discussion] Topology Service losing events when there is port congestion

Jon, Allan,

I just found the following exchange from two years ago and if I understand things correctly, then the
connection between topology services is also governed by the TIPC_FLOW_CONTROL_WIN value which means
that deadlocking is a real issue if I change this value and we're in an upgrade situation.  If this is the
case, then what choice do we have?  In the email exchange 2 years ago you suggested the congestion_limit
addition to the port structure and have it be configurable via setsockopt.  Is there any other simpler
approach you can think of?

Thanks,
Felix

>>-----Original Message-----
>>From: Stephens, Allan [mailto:allan.stephens <at> windriver.com]
>>Sent: May-14-09 16:56
>>To: Nayman Felix-QA5535; Jon Maloy; Natarajan Ramesh-A17988;
>>tipc-discussion <at> lists.sourceforge.net
>>Subject: RE: [tipc-discussion] missing updates from topology service
>>
>>I also think Jon is on to something here.  At first glance, it sounds
>>like we want to introduce the following two socket options:
>>
>>TIPC_RXQ_SIZE: puts an upper bound on the number of queued messages,
>>after which message rejection can occur.  If we set the default to
>>TIPC_OVERLOAD_BASE/2 (i.e. 2500 messages for lowest importance
>>messages), we maintain compatability with existing behavior.

[jon] That was what I had in mind.

(However, maybe this number is way too large?)

[jon] I am not so sure. Since we hear about people running into this limit, and wanting to change it, I think it
is appropriate.

>>
>>TIPC_CONN_WINDOW: puts an upper bound on the number of unacknowledged
>>messages that can be sent over a connection before the sender blocks.
>>Again, for compatability we could default it to
>>TIPC_OVERLOAD_BASE/4 (i.e. 50% of the default TIPC_RXQ_SIZE).

[jon] yes.

>>
>>However, I'm not sure that we might not want to have only the first of
>>these, and simply say that setting a socket's rx queue limit
>>automatically sets the associated port's connection window to half
>>that value.  (We probably need to think about this a bit more ...)

[jon] Unfortunately it is the _other_ connection endpoint that needs to know this value, and we cannot
assume that all nodes have the same settings.
This works in the topology server case because it practically always resides on the same node as the
subscriber port.
(It is of course possible to connect to the topology server on a different node, but I have never heard about
anybody doing this. ) But maybe this is too much of an assumtion too...
The best solution would clearly be if we could transfer a port's rx-limit to the other endpoint, as you
rightly suggest below.

>>
>>The other thing we'd probably eventually want to do is to incorporate
>>the port connection window value into TIPC's connection protocol, so
>>that each end of the connection knows when to stop sending messages.
>>It could be a bit tricky to add this into the initial connection setup
>>messages (i.e. SYN and ACK) -- it might be better to put them into the
>>periodic CONN_MANAGER messages, since this would even potentially open
>>the rode to having the connection window adjusted on-the-fly.

[jon] This is an excellent idea. By forcing a CONN_MANAGER message to be sent immediately after connection
establishment (maybe it is like that now, I don't remember), and otherwise if there is a reason for
adjusting the window size, that would probably solve our problems.

>>> -----Original Message-----
>>> From: Jon Maloy [mailto:jon.maloy <at> ericsson.com]
>>> Sent: Thursday, May 14, 2009 9:43 AM
>>> To: Stephens, Allan; Nayman Felix-QA5535; Natarajan Ramesh-A17988;
>>> tipc-discussion <at> lists.sourceforge.net
>>> Subject: RE: [tipc-discussion] missing updates from topology service
>>>
>>> Hi,
>>> I agree with this, and I think it is possible to come up with a
>>> solution that is generic, i.e. generally usable by all, and
>>which can
>>> easily be applied to the topolgy service to make it more resilient,
>>> without having to introduce any new queues or mechanisms.
>>>
>>> The check for connection congestion at tipc_send() looks af follows:
>>>
>>> static inline
>>> int tipc_port_congested(struct port *p_ptr) {
>>>         return((p_ptr->sent - p_ptr->acked) >=
>>(TIPC_FLOW_CONTROL_WIN
>>> * 2)); }
>>>
>>> Now, we don't want to change TIPC_FLOW_CONTROL_WIN, because of the
>>> risk of misconfiguration, if it becomes different on
>>different nodes,
>>> and because it would affect _all_ connections on that node.
>>> A better approach would be to replace the right part of this
>>> comparison with a configurable integer field in struct port, e.g.
>>> called 'congestion_limit', of course with the default value of
>>> (TIPC_FLOW_CONTROL_WIN * 2), as it is now.
>>> This value for congestion_limit should be configurable via
>>> setscokopt(), just as we should do with the value for
>>> OVERLOAD_LIMIT_BASE.
>>> The toplogy server sets its own value for 'congestion_limit'
>>> at all its port at connection time, either fetched from a new
>>> configuration macro, or simply by using OVERLOAD_LIMIT_BASE.
>>> The latter has the advantage that the connection really will be
>>> aborted before we start to loose topology events.
>>>
>>> Any thought about this?
>>>
>>> Regards
>>> ///jon

-----Original Message-----
From: Felix Nayman
Sent: Thursday, June 02, 2011 2:05 PM
To: 'Jon Maloy'; Stephens, Allan; tipc-discussion <at> lists.sourceforge.net
Subject: RE: [tipc-discussion] Topology Service losing events when there is port congestion

So Jon I have a question about something you said below that's a bit confusing to me.  When you create a
connection to the topology service, isn't that connection a local connection on the same node?  I know that
name table updates are sent across nodes by the node on which the socket is created, but this is handled by
the tipc kernel code running on each node, correct?  And then the local topology service handles the
sending of publish and withdraw messages to the subscribing process, if I'm not mistaken.

If it is local like I thought then there isn't an issue with a mis-match of TIPC versions that can create
deadlock.  We do not have any connection-oriented sockets in our products except for the topology service
connection, therefore I don't think that the deadlock situation you describe is an issue for us.

Thanks,
Felix

-----Original Message-----
From: Jon Maloy [mailto:jon.maloy <at> ericsson.com]
Sent: Friday, May 27, 2011 7:47 PM
To: Stephens, Allan; Felix Nayman; tipc-discussion <at> lists.sourceforge.net
Subject: RE: [tipc-discussion] Topology Service losing events when there is port congestion

Allan,Felix,

First, a comment about importance and connections that we discussed earlier this week:

It is true that the server port inherits the importance set at the client port when using the tcp-connection
style that most people are using now.
But, it is still fully possible to set the importance individually per port, and hence per connection
direction. If you look at how the server port is created in the topology server, (using the remnants of the
native interface, I assume this has not changed), nothing stops the topology server from using a
different importance than the one inherited from the SYN message.
My suggestion is that the topolgy server always should set this importance to at least
TIPC_HIGH_IMPORTANCE, if not even TIPC_CRITICAL_IMPORTANCE for his port, while the client port
continues using the original importance, as set by the user. This does not take any extra resources with
normal usage. I assume that there potentially is much more traffic in the top_srv-to-socket direction
than vice versa, and that it is here the risk of congestion occurs.

My reasoning here is that the topology server, using the native interface, doesn't have the elasticity to
cope with connection congestion, the way a socket has. The topology server can not sleep.
This would dramatically reduce the risk that an event message from the topology server to a socket gets
rejected, but it would not eliminate it. To achieve full elimination of the risk, we would need to use some
sort of out-queue in the topology server, but the question is if that is worthwhile doing.

Of course, this discussion is only relevant if we increase TIPC_FLOW_CONTROL_WIN, which anyway defines
the limit now, to something bigger than 512.

I cannot really remember why I chose exactly that value, but it was more motivated by the wish to not create
too much messaging overhead than by the concern for the socket queue length. It was also hard to see why any
sane traffic excange should permit more than 1024 outstanding messages per direction, and I think I also
implicitly assumed that the sending endpoint would be able to handle such a supposedly rare situation.
With the socket this is no problem, as we know.

Intuitively, the optimal value would be to set it to OVERLOAD_LIMIT_BASE/4, which would mean that the
sending port would register overload and stop sending just before the receiver socket reaches its local
rejection limit. However, OVERLOAD_LIMIT_BASE can be set individually per node, or even per socket
without any consequences the protocol, although we don't do this today. TIPC_FLOW_CONTROL_WIN is
different, since it must have the similar values (but not necessarily the same) at both ends of a
connection. (Imagine one endpoint expecting acks every 512 messages, and stopping sending at 1012
outstanding messages, while the other end thinks it is ok to send an ack just every 3000 message. Complete
deadlock would occur.)

So, what is a good solution? I think it is risky to just change the acknowledge send frequency to anything
bigger, since you cannot assume that the other endpoint of the connection is running the same version of
TIPC. What we can do, is to change the condition for when a sender decides there is overlaod. If we want to do
this, we would have to introduce another endpoint-to-endpoint negotiation, with built-in backwards
compatibility. This would probably be the best solution, since we really could tie the values to OVERLOAD_LIMIT_BASE.

Maybe:
static inline int tipc_port_congested(struct port *p_ptr) {
        return((p_ptr->sent - p_ptr->acked) >= p_ptr->overload_limit_base_for_peer_port));
}

A quick fix for Felix could possibly be:

static inline int tipc_port_congested(struct port *p_ptr) {
        return((p_ptr->sent - p_ptr->acked) >= ((TIPC_FLOW_CONTROL_WIN * 2) +
p_ptr->topology_server_set_adjustment_constant /* 0 for all other ports */ )); }

Have a nice weekend
///jon

Jon Maloy M.Sc. EE
Researcher
Ericsson Canada
Broadband and Systems Research
8400 Decarie
H4P 2N2, Montreal, Quebec, Canada
Phone + 1 514 345-7900 x42056
Mobile + 1 514 591-5578
jon.maloy <at> ericsson.com
www.ericsson.com

-----Original Message-----
From: Stephens, Allan [mailto:allan.stephens <at> windriver.com]
Sent: May-26-11 09:51
To: Felix Nayman; Jon Maloy; tipc-discussion <at> lists.sourceforge.net
Subject: RE: [tipc-discussion] Topology Service losing events when there is port congestion

Felix wrote:

> So we were wondering why the original value of 512 was so low?  I'm
> assuming that the reason was because for connection-oriented sockets
> you wanted to have port congestion kick in first before socket
> congestion as is described in the TIPC Programmer's Guide, but you
> could have set it to 1000 and still been below the 2500 socket congestion limit.

Jon would be the best person to comment on this. I'm afraid the choice of values predates my involvement with TIPC.

> Is there any drawback to setting the value to 10,000 for example?  I
> assume that more memory would be used, but beyond that I don't see any
> other drawbacks.  The sending of the acknowledgment is dependent on
> this value, but waiting to send it until the 10,000th message
> shouldn't create an issue.  Another reason why I believe we can do
> this is because all of our sockets are created with either HIGH
> IMPORTANCE or CRITICAL IMPORTANCE so we will never reach socket
> congestion.  In addition, we do not have any other connection-oriented
> sockets in our product other than the topology service socket.

As far as I can tell, setting the window size to 10K means that you could end up with 20K messages being queued
at the receiving end of a connection before flow control kicks in, which is quite a large number. Whether or
not Linux's sk_buff system can handle this kind of load would determine if this approach would work for you
-- I can't see any issues in TIPC itself that would cause problems. (I *can* say that things would
definitely break down in TIPC if you tried increasing the window size to something in the 32K to 64K range,
as some message fields are limited to 16 bits.)

-- Al

> -----Original Message-----
> From: Stephens, Allan [mailto:allan.stephens <at> windriver.com]
> Sent: Tuesday, May 24, 2011 3:45 PM
> To: Jon Maloy; Felix Nayman; tipc-discussion <at> lists.sourceforge.net
> Subject: RE: [tipc-discussion] Topology Service losing events when
> there is port congestion
>
> Hi Felix:
>
> In my previous email I should have explicitly pointed out a corollary
> to Jon's observation about the use of differing message importance levels:
>
> Instead of increasing the value of OVERLOAD_LIMIT_BASE in your system,
> you could just modify your application to set the subscribing socket
> to TIPC_HIGH_IMPORTANCE before connecting to the topology service.
> (This assumes that you're currently using the default importance level
> of
> TIPC_LOW_IMPORTANCE.)
>
> This change would avoid any problems involving the maximum number of
> messages the port has queued up, without increasing the number of
> messages other TIPC sockets can queue up. It's probably also the more
> desirable approach, since it reduces the changes you have to make to
> TIPC itself to simply changing the value of TIPC_FLOW_CONTROL_WIN.
>
> Regards,
> Al
>
> > -----Original Message-----
> > From: Stephens, Allan
> > Sent: Tuesday, May 24, 2011 3:26 PM
> > To: 'Jon Maloy'; Felix Nayman; tipc-discussion <at> lists.sourceforge.net
> > Subject: RE: [tipc-discussion] Topology Service losing events when
> > there is port congestion
> >
> > Jon wrote:
> >
> > > I believe this is about the connection between the topology
> > > service and user land, so it should always have HIGH_IMPORTANCE in
> > > direction topology-server -> user.
> > > Since importance can be set individually per direction it is the
> > > topology server that should set this value, and without having the
> > > code in front of me I think it already does.
> >
> > This appears to have been true at one point (circa TIPC 1.3), but is
> > no longer true in later releases (TIPC 1.5+). Apparently, when we
> > switched over from the ioctl-based subscription model to the
> > connection-based one we arranged things so that the importance of
> > the connection was determined by the connection originator and
> > TIPC's topology server automatically matches this. This approach
> > still makes sense to me, as it allows us to have connections to the
> > topology service at any importance level, rather than forcing it to
> > a specific
> level.
> >
> > Regards,
> > Al
> >
> > > -----Original Message-----
> > > From: Stephens, Allan [mailto:allan.stephens <at> windriver.com]
> > > Sent: May-24-11 14:43
> > > To: Felix Nayman; tipc-discussion <at> lists.sourceforge.net
> > > Subject: Re: [tipc-discussion] Topology Service losing events when
> > > there is port congestion
> > >
> > > Felix wrote:
> > >
> > > > So we're looking into updating the TIPC_FLOW_CONTROL_WIN from
> > > > 512 to our current candidate of 3000  to avoid the port
> > > > congestion we're seeing on the topology service connection.  So
> > > > I have a number of questions before we do this:
> > > >
> > > > 1)From looking at the code it appears that TIPC_FLOW_CONTROL_WIN
> > > > is used for two purposes:
> > > >         a)to determine if a port is congested
> > > >         b)to determine when to send back an acknowledgment
> > > >
> > > > If we increase the value from 512 to 3000 what side effects do you
> > > > believe we can expect?      Is there also a timer running to
> > determine
> > > > when to send back an acknowledgment or is it only based on the
> > > > number of messages received?
> > >
> > > Increasing TIPC_FLOW_CONTROL_WIN to 3000 means that a connected
> > > port can send a burst of up to 3000 messages to a peer that has no
> > > queued messages before the sender is forced to stop because of
> > > peer
> > congestion.
> > > (Depending on how quickly the peer is able to process these
> > > messages the sending port may actually be able to send up to 6000
> > > messages before it is forced to stop.)
> > >
> > > The acknowledgement mechanism is totally based on the number of
> > > messages processed by the receiver at this time, so
> > > acknowledgements should occur every TIPC_FLOW_CONTROL_WIN messages
> > > regardless of when the processing occurs.
> > >
> > >
> > > > 2)We do not have any other connection-oriented TIPC sockets in
> > > > our system. Does this TIPC_FLOW_CONTROL_WIN only effect
> > > > connection-oriented sockets or connectionless also?
> > >
> > > Connectionless sockets are unaffected.
> > >
> > >
> > > > 3)In the previous discussion you mentioned that we might want to
> > > > consider increasing the OVERLOAD_LIMIT_BASE constant as well.
> > > > What was the reason for this?  If the TIPC_FLOW_CONTROL_WIN is
> > > > still less than the OVERLOAD_LIMIT_BASE we should not reach
> > > > socket congestion,
> > > correct?
> > >
> > > An individual socket is only allowed to queue up
> > > OVERLOAD_LIMIT_BASE/2 messages of the lowest importance level
> > > before it starts reject them, which means 2500 messages in standard TIPC.
> > > So, if you increase TIPC_FLOW_CONTROL_WIN to 3000 you run the risk
> > > that the receiving socket will start rejecting messages before the
> > > sender starts throttling back its sends.
> > >
> > > Note 1: If you're sending messages at TIPC_MEDIUM_IMPORTANCE or
> > > higher, then the receiver allows more messages to be queued up
> > > before it starts rejecting messages. In TIPC 1.7.7 the queue sizes
> > > are allowed to be 5000/250000/infinite for medium/high/critical
> > > importance traffic, respectively. So if you're using elevated
> > > importance you
> > > *might* be able to avoid tinkering with OVERLOAD_LIMIT_BASE, but
> > > see the following note ...
> > >
> > > Note 2: TIPC also imposes a limit of OVERLOAD_LIMIT_BASE for the
> > > total number of messages that can be queued across *all* of its
> > > sockets on any given node. Once this limit is reached TIPC starts
> > > rejecting incoming messages even if the socket that was to receive
> > > the message hasn't yet reached its individual limit.
> > >
> > > > Thanks for all your help,
> > > >
> > > > Felix
> > >
> > >
> > > To summarize with an example:
> > >
> > > If you're using only low importance messages, and you're trying to
> > > send a burst of messages (Mb) on a single connected socket without
> > > blocking the sender, then you need to calculate the maximum number
> > > of messages that you need to be able to queue on that socket (Mi)
> > > and across all sockets (Mt), and ensure that the following
> > > relationships
> > hold:
> > >
> > > a) OVERLOAD_LIMIT_BASE >= Mt
> > > b) OVERLOAD_LIMIT_BASE/2 >= Mi
> > > c) TIPC_FLOW_CONTROL_WIN >= Mb
> > >
> > > where Mb <= Mi <= Mt.
> > >
> > > So, if you are trying to send a burst of 3000 messages to a single
> > > socket that isn't receiving any other traffic (meaning Mb = Mi =
> > > Mt), then setting OVERLOAD_LIMIT_BASE to 6000 and
> > > TIPC_FLOW_CONTROL_WIN to 3000 should get the job done.
> > >
> > > Regards,
> > > Al
> > >
> > > >
> > > > -----Original Message-----
> > > > From: Stephens, Allan [mailto:allan.stephens <at> windriver.com]
> > > > Sent: Tuesday, May 24, 2011 9:04 AM
> > > > To: Felix Nayman; tipc-discussion <at> lists.sourceforge.net
> > > > Subject: RE: [tipc-discussion] Topology Service losing events
> > > > when there is port congestion
> > > >
> > > > Felix wrote:
> > > >
> > > > > So we upgraded to TIPC 1.7.7 and we're noticing port
> > > > > congestion on the topology service socket again.  This is the
> > > > > same issue that we saw about two years ago and that I raised a bug for:
> > > > >
> > > > > http://sourceforge.net/tracker/?func=detail&aid=2790140&group_
> > > > > id
> > > > > =1
> > > > > 65
> > > > > 20
> > > > > &a
> > > > > tid=116520
> > > > >
> > > > > Has anything been done in TIPC 1.7.7 to improve this situation?
> > > > > A number of things were discussed in the email chain we had
> > > > > two years
> > > > ago.
> > > >
> > > > No changes have been made since the last time this issue was
> > > discussed.
> > > > (Development effort has been concentrated on fixing issues with
> > > > TIPC's broadcast link and name table, which are hopefully now
> > > > resolved, and with porting TIPC 1.7 changes into the Linux
> > > > mainstream.)
> > > >
> > > > > If we were able to raise the port congestion limit to 3000 for
> > > > > example from 512, then we would not run into this problem.  Is
> > > > > there a way for us to change compile time constants  such as
> > > > > TIPC_FLOW_CONTROL_WIN and OVERLOAD_LIMIT_BASE to increase this
> > > limit?
> > > >
> > > > At the present time the only way to change these constants is to
> > > > actually modify constants themselves and recompile TIPC.
> > > >
> > > > > Thanks,
> > > > > Felix
> > > >
> > > > Regards,
> > > > Al
> > > >
> > > > ________________________________________________________________
> > > > __ __ __ __
> > > > ______________________________________________________
> > > >
> > > > This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION
> > > > intended solely for the use of the addressee(s). If you are not
> > > > the intended recipient, please notify so to the sender by e-mail
> > > > and delete the original message. In such cases, please notify us
> > > > immediately at sanchar-sadhan <at> infinite.com . Further, you are
> > > > not to copy, disclose, or distribute this e-mail or its contents
> > > > to any unauthorized
> > > > person(s) .Any such actions are considered unlawful. This e-mail
> > > > may contain viruses. Infinite has taken every reasonable
> > > > precaution to minimize this risk, but is not liable for any
> > > > damage you may sustain as a result of any virus in this e-mail.
> > > > You should carry out your own virus checks before opening the
> > > > e-mail or attachments. Infinite reserves the right to monitor
> > > > and review the content of all messages sent to or from this e- mail address.
> > > > Messages sent to or from this e-mail address may be stored on
> > > > the
> Infinite e-mail system.
> > > >
> > > > ***INFINITE******** End of Disclaimer********INFINITE********
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > --
> > > --
> > > ------
> > > vRanger cuts backup time in half-while increasing security.
> > > With the market-leading solution for virtual backup and recovery,
> > > you get blazing-fast, flexible, and affordable data protection.
> > > Download your free trial now.
> > > http://p.sf.net/sfu/quest-d2dcopy1
> > > _______________________________________________
> > > tipc-discussion mailing list
> > > tipc-discussion <at> lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/tipc-discussion
> ______________________________________________________________________
> __ ______________________________________________________
>
> This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended
> solely for the use of the addressee(s). If you are not the intended
> recipient, please notify so to the sender by e-mail and delete the
> original message. In such cases, please notify us immediately at
> sanchar-sadhan <at> infinite.com . Further, you are not to copy, disclose,
> or distribute this e-mail or its contents to any unauthorized
> person(s) .Any such actions are considered unlawful. This e-mail may
> contain viruses. Infinite has taken every reasonable precaution to
> minimize this risk, but is not liable for any damage you may sustain
> as a result of any virus in this e-mail. You should carry out your own
> virus checks before opening the e-mail or attachments. Infinite
> reserves the right to monitor and review the content of all messages
> sent to or from this e- mail address. Messages sent to or from this
> e-mail address may be stored on the Infinite e-mail system.
>
> ***INFINITE******** End of Disclaimer********INFINITE********

______________________________________________________________________________________________________________________________

This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the
addressee(s). If you are not the intended recipient, please notify so to the sender by e-mail and delete
the original message. In such cases, please notify us immediately at sanchar-sadhan <at> infinite.com .
Further, you are not to copy, disclose, or distribute this e-mail or its contents to any unauthorized
person(s) .Any such actions are considered unlawful. This e-mail may contain viruses. Infinite has
taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain
as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail
or attachments. Infinite reserves the right to monitor and review the content of al
 l messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on
the Infinite e-mail system.

***INFINITE******** End of Disclaimer********INFINITE********

______________________________________________________________________________________________________________________________

This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the
addressee(s). If you are not the intended recipient, please notify so to the sender by e-mail and delete
the original message. In such cases, please notify us immediately at sanchar-sadhan <at> infinite.com .
Further, you are not to copy, disclose, or distribute this e-mail or its contents to any unauthorized
person(s) .Any such actions are considered unlawful. This e-mail may contain viruses. Infinite has
taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain
as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail
or attachments. Infinite reserves the right to monitor and review the content of al
 l messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on
the Infinite e-mail system. 

***INFINITE******** End of Disclaimer********INFINITE******** 

------------------------------------------------------------------------------
Get a FREE DOWNLOAD! and learn more about uberSVN rich system, 
user administration capabilities and model configuration. Take 
the hassle out of deploying and managing Subversion and the 
tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2