Share data between services

Go To


I need to break up my duplex service and would like to encapsulate the large transfers into one service and retrieve from other(s). I used to have it all in one service but now need to switch from buffered to streaming to account for size/memory accommodations. I have seen a few questions here and here but they are quite old

What would I use for IPC between services, a namedPipe?

Service A exposes 2 methods Guid Upload(stream),Stream Download(Guid) and uses net.tcp streaming, this is working well,

Now I would like to persist to Service B? Would this be the namedPipe WCF?

Service C --> Business layer --> Service B with Guid, retrieve and do calculations on item, persist back to B

My question is what to use for persistence/Service B

From the clients perspective

  1. Client calls ServiceA_Proxy.Upload(someLargeItem) returns Guid
  2. Client then calls ServiceC_Proxy.DoSomeWork(GuidFromCall_1)
  3. Client then calls ServiceA_Proxy.Download(GuidFromCall_1)
  4. Client displays to enduser
2012-04-03 21:41
by Eric


Yes you can use Named Pipes as a WCF Binding as long as comunicating processes are run on the same server. Basically you create WCF service just as if you were using any other binding like TCP/IP, but you'll need to configure your endpoint to use netNamedPipeBinding.

If you want to persist/save the data then you can save it to files or even database, depending on your requirements. If you just wanted to keep the data until ServiceC calls for it, then Dictionary<Guid, SomeLargeItem> would be enough.

So your flow would look like:

  1. Client calls ServiceA_Proxy.Upload(someLargeItem) returns Guid
  2. ServiceA calls ServiceB_Proxy.Upload(GuidFromCall_1, someLargeItem)
  3. Client then calls ServiceC_Proxy.DoSomeWork(GuidFromCall_1) (you need first to make sure that upload is finished)
  4. ServiceC calls ServiceB_Proxy.Download(GuidFromCall_1)
  5. ServiceC calls DoSomeWork(GuidFromCall_1) (Internally)
  6. ServiceC calls ServiceB_Proxy.Upload(GuidFromCall_1, someLargeItemProcessed)
  7. Client then calls ServiceA_Proxy.Download(GuidFromCall_1) (again, you need to make sure that processing is finished)
  8. ServiceA calls ServiceB_Proxy.Download(GuidFromCall_1), and returns to Client.
  9. Client displays to enduser

I'm not sure I didn't mess something up. As you can see, this is getting pretty complex. You need to ask yourself it this is the way to go and if splitting your service this way will really make it more scalable? You plan to use NamedPipes so all the processing would still be on the same machine.

You should really consider your design. Maybe splitting the funcionality into some *.dlls and calling them directly would be enough? This way you'll achieve logical separation of layers. Switching from buffered to streaming communication doesn't require you to split your logic into more services.

And if you want true scalability, consider using queues (for example MSMQ) and separate machines for each service.

2012-04-03 23:15
by surfen
do have an example on how "Switching from buffered to streaming communication doesn't require you to split your logic into more services" when I need a duplex pipe and streaming? Everything was done in ServiceC, but the uploads are too large and I cannot buffer those so I thought a central cache would be the way to g - Eric 2012-04-04 00:55
How large are your uploads? Is the problem in the way you transfer them or in the way that you keep them in memory after transfer is finished? You're right that it's not possible to use both streaming and duplex, but you should consider redesigning your solution to use request-replay instead of duplex in this case - surfen 2012-04-04 01:17
500MB - 2GB for up/down. When it was one service, I would use callbacks for the upload chunks, processing, and finished and just used a Dictionary for retrieval later. The size was not taken into account when we contracted this out, so it was only tested with > 100MB, now I get OOM exceptions. Was looking for a quick fix but this becoming quite the rucku - Eric 2012-04-04 01:25
What kind of data is that. Maybe compressing it would help - surfen 2012-04-04 01:33
Custom entities, we use ICSharpCode.SharpZipLib.GZip in other places but mostly for file uploads and have not tried with this. Was also thinking of just doing string filePath = System.IO.Path.Combine(Path.GetTempPath(), guidToken); and using the same path in the streamer service and avoid the IPC altogethe - Eric 2012-04-04 01:40
Yes, I think that would be better than creating another service. Just serialize the SomeLargeItem to disk and deserialize it when requested - surfen 2012-04-04 01:46